Changes made in trunk only since 0.8-series is now stable and this version only runs on Python 2 which is end-of-life as of 1/1/2020. 0.7.19 is the last of this series. —————————– * Merged in lp:~mgorse/duplicity/0.7-series - Fix bug 1828869: refresh CollectionsStatus after sync * Merged in lp:~aaron-whitehouse/duplicity/07-snap - Add snapcraft packaging instructions for 0.7 series
--restore-timeoption when using this new option.
Enhancements: * Undid move of testing/test_code.py. Instead I fixed it so that it would not run during PPA build. It now needs the setting RUN_CODE_TESTS=1 in the environment which is supplied in the tox.ini file. * Moved testing/test_code.py to testing/manual/code_test.py so PPA builds would succeed. Should be moved back later. * Remove valid_extension() check from file_naming.py. It was causing failed tests for short filenames. Thanks edso. * Partial fix for PPA build failures, new backend name. * Merged in lp:~ed.so/duplicity/fix.dpbx.import - fix dpbx import error import lazily * Merged in lp:~hooloovoo/duplicity/fix-typo-in-test-description - Fixed spelling mistake/typo in a description of a test. * Merged in lp:~mterry/duplicity/missing-unicode-escape - Convert restore_dir to unicode before printing. * Merged in lp:~ed.so/duplicity/lftp.ncftp.and.prefixes - retire –ssh-backend, –use-scp parameters - introduce scheme prefixes for alternative backend selection e.g. ncftp+ftp://, see manpage - scp is now selected via scheme e.g. scp:// - added lftp fish, webdav(s), sftp support * Merged in lp:~mterry/duplicity/code-nits - Fix some pylint/pep8 nits that prevented the test_code.py test from passing. * Merged in lp:~mterry/duplicity/debian-dir - Add a debian/ directory to make it easier to manage the PPAs for duplicity. * In webdavbackend.py: - Fixed bug 1396106 with change by Tim Ruffing, mispelled member. - Added missing ‘self.’ before member in error message. * Merged in lp:~adrien-delhorme/duplicity/hubic - Add Hubic support through pyrax and a custom pyrax_identity module. * Fixed bug 1385599 with changes by Yannick Molin. SSL settings are now conditioned on protocol ftp or ftps. * Partial fix of bug 1236248 with changes by az, manpage warning about –extra-clean, however, recovery with missing sig files is broken. * Fixed bug 1255453 with changes by Gaudenz Steinlin, report backend import results, both normal and failed, at INFO log level. * Manually merged in lp:~m4ktub/duplicity/0.6-reliability - Per fix proposed in Bug #1395341. * Modded .bzrignore to ignore .egg test dependencies, normalized, sorted. Merged in lp:~ed.so/duplicity/paramiko.identyfile - fix identity file parsing of –ssh-options for paramiko - manpage fixes * Source formatted, using PyDev, all source files to fix some easily fixed PEP8 issues. Use ignore space when comparing against previous versions. * Merge in lp:~andol/duplicity/signkeyformat - Allow –sign-key to use short format, long format alt. full fingerprint. * Merge in lp:~hooloovoo/duplicity/verify-not-check-source - Tests to validate that duplicity does not check filesystem source during verify unless –compare-data is specified * Merge in lp:~ed.so/duplicity/move_netloc - move netloc usage definitions into respective backends - fix “[Question #259173]: rsync backend fails” https://answers.launchpad.net/duplicity/+question/259173 * Make ssh an unsupported backend scheme * Temporarily disable RsyncBackendTest and test_verify_changed_source_file * Merge in lp:~hooloovoo/duplicity/test-verify-improvements - Fix up test_verify, which was a bit of a mess: - Simplify test_verify.py to just do a simple backup and verify on a single file in each test. - Modify tests to correctly use –compare-data option. - Add tests for when the source files have atime/mtime manipulated. * Fix duplicity verify to ignore the file system when globals.compare_data is False. This means that verify only validates the viability of the backup itself unless –compare-data is specified. * Reenable test_verify_changed_source_file test * Merged in lp:~hooloovoo/duplicity/add-additional-verify-tests-for-corrupted-archives - Add tests to test_verify.py to test that verify fails if the archive file is corrupted. Changed file objects to use the with keyword to ensure that the file is properly closed. - Small edit to find statement in verify_test.sh to make it work as expected (enclose string in quotes). * Merged in lp:~hooloovoo/duplicity/add-else-to-badupload-try-except - Badupload test previously did not have an else in the try-except. The test passed if the except was triggered, but would also pass if the test did not trigger an error at all. * Fixed bug 1406173 by applying patch supplied in report - Ignore .par2 files in remote file list * Removed redundant shell test testing/verify_test.sh * Misc fixes for the following PEP8 issues: - E211, E221, E222, E225, E226, E228 - see http://pep8.readthedocs.org * Fixed bug 1278529 by applying patch supplied in report - Use get_bucket() rather than lookup() on S3 to get proper error msg. * Merged in lp:~stapelberg+ubuntu/duplicity/add-onedrive-backend - Add a Microsoft OneDrive backend * Merged in lp:~hooloovoo/duplicity/filelist_select_bug_1408411 - Adds functional test cases that fail because of Bug #1408411 (commented out), to assist in fixing that bug. * Merged in lp:~hooloovoo/duplicity/process_filelists_for_spaces_etc - Process filelists to remove imperfections such as blank lines, comments and leading/trailing whitespace. Also correctly processes quoted folders containing spaces in their names. Extensive unit and functional tests to test these changes (and selection more generally). - The branch does add an additional folder to testfiles.tar.gz called select2. This included a folder with a trailing space, to test the quote test. The subfolders also have clearer names than in the “select” folder (eg “1sub2sub3”) which makes it easier to keep track of issues in tests. * Merged in lp:~9-sa/duplicity/FixBug1408289 - Fix bug #1408289 - Wrong attribute name prevented raise of client exception, working now * Merged in lp:~noizyland/duplicity/azurebackend - Add backend for Azure Blob Storage Service
Enhancements: * Adjust unit tests to expect single FTP backend * Merged in lp:~moritzm/duplicity/duplicity - Use lftp for both FTP and FTPS * Merged in lp:~ed.so/duplicity/0.7-dpbx.importfix - fix this showstopper with the dropbox backend “NameError: global name ‘rest’ is not defined” * Merged in lp:~jflaker/duplicity/BugFix1325215 - The reference to “–progress_rate” in the man page as a parameter is incorrect. Should be “–progress-rate”. * Merged in lp:~hooloovoo/duplicity/updated-README-REPO - Changes to README-REPO to reflect the restructuring of the directories. * Fixed bug 1375304 with patch supplied by Aleksandar Ivanovic * Merged in lp:~ed.so/duplicity/webdav200fix-0.7 - webdav backend fix “BackendException: Bad status code 200 reason OK.” when restarting an interrupted backup and overwriting partially uploaded volumes. * Merged in lp:~mterry/duplicity/require-2.6 - Require at least Python 2.6. - Our code base already requires 2.6, because 2.6-isms have crept in. Usually because we or a contributor didn’t think to test with 2.4. And frankly, I’m not even sure how to test with 2.4 on a modern system. * Merged in lp:~mterry/duplicity/drop-pexpect - Drop our local copy of pexpect in favor of a system version. - It’s only used by the pexpect ssh backend (and if you’re opting into that, you probably can expect that you will need pexpect) and the tests. - I’ve done a quick smoketest (backed up and restored using –ssh-backend=pexpect) and it seemed to work fine with a modern version of pexpect. * Merged in lp:~mterry/duplicity/2.6isms - Here’s a whole stack of minor syntax modernizations that will become necessary in python3. They all work in python2.6. - I’ve added a new test to keep us honest and prevent backsliding on these modernizations. It runs 2to3 and will fail the test if 2to3 finds anything that needs fixing (with a specific set of exceptions carved out). - This branch has most of the easy 2to3 fixes, the ones with obvious and safe syntax changes. - We could just let 2to3 do them for us, but ideally we use 2to3 as little as possible, since it doesn’t always know how to solve a given problem. I will propose a branch later that actually does use 2to3 to generate python3 versions of duplicity if they are requested. But this is a first step to clean up the code base. * Merged in lp:~mterry/duplicity/drop-static - Drop static.py. - This is some of the oldest code in duplicity! A bzr blame says it is unmodified (except for whitespace / comment changes) since revision 1. - But it’s not needed anymore. Not really even since we updated to python2.4, which introduced the @staticmethod decorator. So this branch drops it and its test file. * Merged in lp:~mterry/duplicity/py3-map-filter - In py3, map and filter return iterable objects, not lists. So in each case we use them, I’ve either imported the future version or switched to a list comprehension if we really wanted a list. * Merged in lp:~mterry/duplicity/backend-unification - Reorganize and simplify backend code. Specifically: - Formalize the expected API between backends and duplicity. See the new file duplicity/backends/README for the instructions I’ve given authors. - Add some tests for our backend wrapper class as well as some tests for individual backends. For several backends that have some commands do all the heavy lifting (hsi, tahoe, ftp), I’ve added fake little mock commands so that we can test them locally. This doesn’t truly test our integration with those commands, but at least lets us test the backend glue code. - Removed a lot of duplicate and unused code which backends were using (or not using). This branch drops 700 lines of code (~20%) in duplicity/backends! - Simplified expectations of backends. Our wrapper code now does all the retrying, and all the exception handling. Backends can ‘fire and forget’ trusting our wrappers to give the user a reasonable error message. Obviously, backends can also add more details and make nicer error messages. But they don’t have to. - Separate out the backend classes from our wrapper class. Now there is no possibility of namespace collision. All our API methods use one underscore. Anything else (zero or two underscores) are for the backend class’s use. - Added the concept of a ‘backend prefix’ which is used by par2 and gio backends to provide generic support for “schema+” in urls – like par2+ or gio+. I’ve since marked the ‘–gio’ flag as deprecated, in favor of ‘gio+’. Now you can even nest such backends like par2+gio+file://blah/blah. - The switch to control which cloudfiles backend had a typo. I fixed this, but I’m not sure I should have? If we haven’t had complaints, maybe we can just drop the old backend. - I manually tested all the backends we have (except hsi and tahoe – but those are simple wrappers around commands and I did test those via mocks per above). I also added a bunch more manual backend tests to ./testing/manual/backendtest.py, which can now be run like the above to test all the files you have configured in config.py or you can pass it a URL which it will use for testing (useful for backend authors). * Merged in lp:~mterry/duplicity/py2.6.0 - Support python 2.6.0. - Without this branch, we only support python >= 2.6.5 because that’s when python’s urlparse.py module became its more modern incarnation. (I won’t get into the wisdom of them making such a change in the middle of the 2.6 lifecycle.) - Also, the version of lockfile that I have (0.8) doesn’t work with python 2.6.0 or 2.6.1 due to their implementation of threading.current_thread().ident returning None unexpectedly. So this branch tells lockfile not to worry about adding the current thread’s identifier to the lock filename (we don’t need a separate lock per thread, since our locking is per process). - I’ve tested with 2.6.0 and 2.7.6 (both extremes of our current support). * Update shebang line to python2 instead of python to avoid confusion. * Merged in lp:~3v1n0/duplicity/copy.com-backend - I’ve added a backend for Copy.com cloud storage, this supports all the required operations and works as it should from my tests. - You can use it by calling duplicity with something like: copy://email@example.com:firstname.lastname@example.org/duplicity - The only thing I’ve concerns with is the optimized support for _delete_list which can’t be enabled here because the test_delete_list tries also to delete a not-existing files, and it requires the backend not to raise an exception in that case (is this somewhat wanted or could we do the same as for _delete or _query?) * Merged in lp:~ed.so/duplicity/webdav200fix-0.7 - webdav backend fix “BackendException: Bad status code 200 reason OK.” when restarting an interrupted backup and overwriting partially uploaded volumes. * Merged in lp:~mterry/duplicity/webdav-fixes - This branch fixes two issues I saw when testing the webdav backend: - 1) Errors like the following: “Attempt 1 failed. BackendException: File /tmp/duplicity-LQ1a0i-tempdir/mktemp-u2aiyX-2 not found locally after get from backend”. These were caused by the _get() method not calling setdata() on the local path object, so the rest of the code thought it didn’t exist. - 2) Some odd issues from stale responses/data. We have a couple places in webdavbackend.py where we close the connection before making a request because of this problem. But I’ve changed it to do it every time, more reliably, by putting a _close() call inside the request() method. - With this, the webdav backend seems fine to me. * Merged in lp:~antmak/duplicity/0.7-par2-fix - Useful fix for verbatim par2cmdline options (like “-t” in par2-tbb version) * Fixed bug 1327550: OverflowError: signed integer is greater than maximum - Major and minor device numbers are supposed to be one byte each. Someone has crafted a special system image using OpenVZ where the major and minor device numbers are much larger (ploop devices). We treat them as (0,0). * Added sxbacked.py, Skylable backend. Waiting on man page updates. * Merged in lp:~ed.so/duplicity/manpage.verify - Clarify verify’s functionality as wished for by a user surprised with a big bandwidth bill from rackspace. * Merged in lp:~jeffreydavidrogers/duplicity/duplicity - This change fixes two small typos in the duplicity man page. * Merged in lp:~johnleach/duplicity/1315437-swift-container-create - Check to see if the swift container exists before trying to create it, in case we don’t have permissions to create containers. Fixes #1315437 * Merged in lp:~ed.so/duplicity/manpage.blocksize - add –max_blocksize doc - reorder ‘a note on filename prefixes’ into alphabetical order
Enhancements: * Applied two patches from mailing list message at: https://lists.nongnu.org/archive/html/duplicity-talk/2014-01/msg00030.html “Added command line options to use different prefixes for duplicity files” This resolves https://bugs.launchpad.net/duplicity/+bug/1170161 and provides a workaround for https://bugs.launchpad.net/duplicity/+bug/1170113 * Merged in lp:~mterry/duplicity/gpg-encode - getpass.getpass(prompt) eventually calls str(prompt). Which is a no go, if the prompt contains unicode. Here’s a patch to always pass getpass() a byte string. - Our tests didn’t catch this because they always set PASSPHRASE. I’ve added a test that passes the passphrase via stdin. * Merged in lp:~mterry/duplicity/pexpect-fix - duplicity has its own copy of pexpect. Use that instead of requiring one from the system. * Merged in lp:~prateek/duplicity/s3-glacier - Fixes https://bugs.launchpad.net/duplicity/+bug/1039511 - Adds support to detect when a file is on Glacier and initiates a restore to S3. Also merges overlapping code in the boto backends - Fixes https://bugs.launchpad.net/duplicity/+bug/1243246 - Adds a –s3_multipart_max_timeout input option to limit the max execution time of a chunked upload to S3. Also adds debug message to calculate upload speed. * Merged in lp:~ed.so/duplicity/fix.dpbx - Fix dpbx backend “NameError: global name ‘rest’ is not defined” * Merged in lp:~prateek/duplicity/botoimportfix - Switches the boto backend back to using lazy imports so there are no complaints during the importing of backends. * Merged in lp:~germer/duplicity/par2 - This branch adds Par2 recovery files to duplicity. It is a wrapper backend which will create the recovery files and upload them all together with the wrapped backend. Corrupt archives will be detected and repaired (if possible) on the fly during restore. - It can be used with url-string par2+webdavs://USER@HOST/PATH - Fixes https://bugs.launchpad.net/duplicity/+bug/426282 * Merged in lp:~fredrik-loch/duplicity/duplicity-S3-SSE - Adds support for server side encryption as requested in Bug #996660 * Merged in lp:~mterry/duplicity/drop-u1 - Ubuntu One is closing shop. So no need to support a u1 backend anymore. * Merged in lp:~mterry/duplicity/fix-drop-u1 - Looks like when the drop-u1 branch got merged, its conflict got resolved badly. Here is the right version of backend.py to use (and also drops u1backend.py from POTFILES). * Merged in lp:~mterry/duplicity/drop-pexpect - Drop our local copy of pexpect in favor of a system version. - It’s only used by the pexpect ssh backend (and if you’re opting into that, you probably can expect that you will need pexpect) and the tests. - I’ve done a quick smoketest (backed up and restored using –ssh-backend=pexpect) and it seemed to work fine with a modern version of pexpect. * Merged in lp:~mterry/duplicity/2.6isms - Here’s a whole stack of minor syntax modernizations that will become necessary in python3. They all work in python2.6. - I’ve added a new test to keep us honest and prevent backsliding on these modernizations. It runs 2to3 and will fail the test if 2to3 finds anything that needs fixing (with a specific set of exceptions carved out). - This branch has most of the easy 2to3 fixes, the ones with obvious and safe syntax changes. - We could just let 2to3 do them for us, but ideally we use 2to3 as little as possible, since it doesn’t always know how to solve a given problem. I will propose a branch later that actually does use 2to3 to generate python3 versions of duplicity if they are requested. But this is a first step to clean up the code base. * Merged in lp:~mterry/duplicity/drop-static - Drop static.py. - This is some of the oldest code in duplicity! A bzr blame says it is unmodified (except for whitespace / comment changes) since revision 1. - But it’s not needed anymore. Not really even since we updated to python2.4, which introduced the @staticmethod decorator. So this branch drops it and its test file. * Merged in lp:~mterry/duplicity/encode-for-print - Encode translated strings before passing them to ‘print’. - The print command can only apparently handle bytes. So when we pass it unicode, it freaks out. There were only four instances I saw where we used print, so I figured it was easiest to just convert them to use the log framework too. - That way all user-visible strings go through that framework and are subject to the same encoding rules. * Merged in lp:~mterry/duplicity/more-test-reorg - Here’s another test reorganization / modernization branch. It does the following things: - Drop duplicity/misc.py. It is confusing to have both misc.py and util.py, and most of the code in misc.py was no longer used. I moved the one function that was still used into util.py. - Consolidated the various ways to run tests into just one. I made tox runs go through ./setup.py test, rather than nosetests. And I made the ./testing/run-tests scripts just call tox. Now we no longer need nosetests as a test dependency (although you can still use it if you want). - Added two more code quality automated tests: a pep8 one and a pylint one. I disabled almost all checks in each program that gave a warning. These tests just establish a baseline for future improvement. - Moved the test helper code into TestCase subclasses that all tests can use. And used more code sharing and setUp/tearDown cleverness to remove duplicated code. - Reorganized the tests in ./testing/tests into ./testing/functional and ./testing/unit – for whether they drive duplicity as a subprocess or whether they import and test code directly. Each dir can have specialized TestCase subclasses now. - Renamed the files in ./testing/unit to more clearly indicate which file in ./duplicity they are unit testing. - Added some helper methods for tests to set environment and globals. parameters more safely (i.e. without affecting other tests) by automatically cleaning up any such changes during test tearDown. - Removed test_unicode.py, since it is kind of dumb. It used to be more useful, but now with py2.6, we are just testing that one line of code in it is actually there. Fixed bug #1312328 WebDAV backend can’t understand 200 OK response to DELETE - Allow both 200 and 204 as valid response to delete * Merged in lp:~mterry/duplicity/py3-map-filter - In py3, map and filter return iterable objects, not lists. So in each case we use them, I’ve either imported the future version or switched to a list comprehension if we really wanted a list. * Merged in lp:~mterry/duplicity/backend-unification - Reorganize and simplify backend code. Specifically: - Formalize the expected API between backends and duplicity. See the new file duplicity/backends/README for the instructions I’ve given authors. - Add some tests for our backend wrapper class as well as some tests for individual backends. For several backends that have some commands do all the heavy lifting (hsi, tahoe, ftp), I’ve added fake little mock commands so that we can test them locally. This doesn’t truly test our integration with those commands, but at least lets us test the backend glue code. - Removed a lot of duplicate and unused code which backends were using (or not using). This branch drops 700 lines of code (~20%) in duplicity/backends! - Simplified expectations of backends. Our wrapper code now does all the retrying, and all the exception handling. Backends can ‘fire and forget’ trusting our wrappers to give the user a reasonable error message. Obviously, backends can also add more details and make nicer error messages. But they don’t have to. - Separate out the backend classes from our wrapper class. Now there is no possibility of namespace collision. All our API methods use one underscore. Anything else (zero or two underscores) are for the backend class’s use. - Added the concept of a ‘backend prefix’ which is used by par2 and gio backends to provide generic support for “schema+” in urls – like par2+ or gio+. I’ve since marked the ‘–gio’ flag as deprecated, in favor of ‘gio+’. Now you can even nest such backends like par2+gio+file://blah/blah. - The switch to control which cloudfiles backend had a typo. I fixed this, but I’m not sure I should have? If we haven’t had complaints, maybe we can just drop the old backend. - I manually tested all the backends we have (except hsi and tahoe – but those are simple wrappers around commands and I did test those via mocks per above). I also added a bunch more manual backend tests to ./testing/manual/backendtest.py, which can now be run like the above to test all the files you have configured in config.py or you can pass it a URL which it will use for testing (useful for backend authors). * Merged in lp:~mterry/duplicity/encode-exceptions - Because exceptions often contain file paths, they have the same problem with Python 2.x’s implicit decoding using the ‘ascii’ encoding that we’ve experienced before. So I added a new util.uexc() method that uses the util.ufn() method to convert an exception to a unicode string and used it around the place. - Bugs fixed: 1289288, 1311176, 1313966 * Applied expat fix from edso. See answer #12 in https://answers.launchpad.net/duplicity/+question/248020
Enhancements: * Applied patch from bug 1216921 to fix ignore_missing(). - merged lp:~mterry/duplicity/ignore-missing to fix patch. * Merged in lp:~mterry/duplicity/catch-seq-copy-error - Any* exception when running patch_seq2ropath should be ignored (though logged) and duplicity should move on. This covers the two asserts in that function (bug 1155345 and bug 720525) as well as errors that happen during file copying (bug 662442). * Merged in lp:~mterry/duplicity/argv - Fix use of argv when calling os.execve * Merged in lp:~verb/duplicity/bucket_root_fix - Fix bug that prevents backing up to the root of a bucket with boto backend. * Merged in lp:~gliptak/duplicity/415619 - Better error message when chown fails * Merged in lp:~mterry/duplicity/log-path-type - Any backup browser built on top of duplicity will need to indicate which files in the backup are folders and which are files. The current logging information doesn’t provide this detail. So I’ve added a field to the log.InfoCode.file_list output that includes the path type. * Merged in lp:~mterry/duplicity/manifest-oddities - We may accidentally end up with an oddly inconsistent manifest like so: Volume 1 Volume 2 Volume 3 Volume 2 As did get reported recently on the mailing list: http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00009.html - One way this can happen (the only way?) is if you back up, then duplicity gets interrupted between writing the manifest and uploading the volume. Then, when restarted, there is no longer enough data to create as many volumes as existed previously. - This situation can cause an exception when trying to restart the backup. - This branch fixes it by deleting any excess volume information encountered when loading in the manifest. We discard volume with higher numbers than the last one read. * Merged in lp:~mterry/duplicity/disappearing-source - When restarting a backup, we may accidentally skip the first chunk of one of the source files. To reproduce this,: 1) interrupt a backup 2) delete the source file it was in the middle of 3) restart the backup - When replaying the source iterator to find where to resume from, we can’t notice that the file is gone until we’ve already iterated past where it would be! - The solution I came up with is to just let duplicity stuff the data we accidentally read back into the source iterator. - This is actually a data loss bug, because it’s possible to back up corrupted files (that are missing their first chunk). * Merged in lp:~mterry/duplicity/normalize-before-using - Avoid throwing an exception due to a None element in a patch sequence. - None elements in a (non-normalized) patch sequence are perfectly normal. With the current code in the patched function, it is certainly possible to hit a crash due a None. See http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00005.html - This branch fixes that by normalizing the sequence before using it in the logging code. It’s acceptable to bring the normalize_ps() call outside the try/except block because normalize_ps is not expected to throw. It’s relatively simple and doesn’t really use its objects besides checking if they are None. * Applied patch to fix “Access GDrive through gdocs backend failing” - see https://lists.nongnu.org/archive/html/duplicity-talk/2013-07/msg00007.html * Merged in lp:~jkrauss/duplicity/pyrax - Rackspace has deprecated python-cloudfiles in favor of their pyrax library, which consolidates all Rackspace Cloud API functionality into a single library. Tested it with Duplicity 0.6.21 on both Arch Linux and FreeBSD 8.3.0. * Changed to default to pyrax backend rather than cloudfiles backend. To revert to the cloudfiles backend use ‘–cf-backend=cloudfiles’ * Merged in lp:~verb/duplicity/boto-min-version - Update documentation and error messages to match the current actual version requirements of boto backend. * Merged in lp:~ed.so/duplicity/debian.paramiko.log - upstream debian patch “paramiko logging” http://patch-tracker.debian.org/package/duplicity/0.6.22-2 * Merged in lp:~ed.so/duplicity/debian.dav.mkdir - upstream debian patch “webdav create folder recursively” http://patch-tracker.debian.org/package/duplicity/0.6.22-2 * Nuke tabs * Merged in lp:~mterry/duplicity/encoding - This branch hopefully fixes two filename encoding issues: - Users in bug 989496 were noticing a UnicodeEncodeError exception which happens (as far as I can tell) because some backends (like webdav) are returning unicode filenames from list(). When these filenames are combined with the utf8 translations of log messages, either (A) the default ascii encoding can’t handle promoting the utf8 bytes or – if there aren’t any utf8 bytes in the translation – (B) the resulting unicode string raises an error later when log.py tries to upgrade the string again to unicode for printing. - This fix is largely implemented by adding a wrapper for backend list() implementations. This wrapper ensures that duplicity internals always see a byte string. (I’d like to eventually use this same wrapping strategy to implement generic retry support without backends having to add any logic, but that’s just a thought for the future.) - That is, the fix for issue #1 is completely inside backend.py and the changes to backends/.py. - The rest of the invasive changes deal with filenames that may not be valid utf8. This is much rarer, but possible. For proper handling of this, we need to print using unicode, and convert filenames from the system filename encoding to unicode, gracefully handling conversion errors. Some of the filenames we print are remote names. Who knows what encoding they are in; it could be different than the system filename encoding. 99% of the time, everything will be utf8 and we’re fine. If we do get conversion errors, the only effect should be some question mark characters in duplicity logging output. - I tried to convert as much of the actual codebase to use unicode for printing. But I stopped short of adding an assert in log.py to enforce unicode, because I didn’t want to go through all the backend code and manually adjust those bits without being able to test each one. Restored missing line from patch of gdocsbackend.py * Reverted changes to gdocsbackend.py * Restored patch of gdocsbackend.py from original author (thanks ede) * Applied patch from bug 1266753: Boto backend removes local cache if connection cannot be made * Merged in lp:~louis-bouchard/duplicity/add-allow-concurrency - Implement locking mechanism to avoid concurrent execution under the same cache directory. This is the default behavior. - Also implement –alllow-concurrency option to disable the locking if required. - This functionality adds a dependency to python-lockfile
Enhancements: * Applied patches from Laszlo Ersek to rdiffdir to “consume a chain of sigtar files in rdiffdir delta mode” which supports incremental sigtar files. * Merged in lp:~jnoster/duplicity/dpbx-added - Add Dropbox backend - NB! In order to use the backend one must: 1. Install Dropbox Python SDK first. 2. Run the duplicity with Dropbox backend (dpbx://) first time interactively to catch and follow the oAuth URL. * Merged in lp:~ed.so/duplicity/verify.data - add switch –compare-data, to selectively enable formerly always disabled data comparison on verify runs * Merged in lp:~tblue/duplicity/paramiko-1.10.0 - This fixes bug #1156746, making the Paramiko backend compatible with Paramiko 1.10.0. It keeps compatibility with older Paramiko versions. * Merged in lp:~townsend/duplicity/fix-1161599-2 - The fix in revno. 912 didn’t take into account that the parameter “body” passed into request() is overloaded, so when it was NULL or of a type other than file, it would fail. This checks if “body” is of type “file” before actually seek()’ing back to the beginning of the file. * Merged in lp:~tblue/duplicity/paramiko-fix-delete-retry - This fixes bug #1115715, which is really annoying. Basically it makes using the Paramiko backend with the default settings impossible. * Merged in lp:~juan-f/duplicity/progress - From time ago, there are people asking for a progress bar estimation in duplicity. There is even a script that circumvents the issue, getting info from the log so as to estimate the progress status ( https://github.com/quentin/Duplicity-progress ) but does not give enough feedback and the estimation is rather plain. - I have developed a set of heuristics that gather information from the deltas and the transfer ratios of the backend so as to forecast % of progress, estimation of remaining time and average speed, for both full and incremental backup uploads. - The current implementation works for boto backend, but to port the other backends to use this feature would be quite easy (we can discuss the details if interested). - The algorithm is activated by the –progress command line flag, and will perform a first-pass dry-run to collect evidence for all the deltas. Next it will trigger the real upload, while a thread statistically estimates the ratio of changes and compression for the data in/out, and uses these ratios to forecast time remaining and % of completion. - The progress data will be logged each 3 seconds, or the –progress-rate flag. * Merged in lp:~jnoster/duplicity/dpbx-added - The application key was approved as “production” one after some changes to the code to suit the requirements of Dropbox team (the keys are now obfuscated, for instance). * Applied blocksize.patch from https://bugs.launchpad.net/duplicity/+bug/897423 - New option –max-blocksize (default 2048) to allow increasing delta blocksize. * Applied duplicity-ftps.patch from https://bugs.launchpad.net/duplicity/+bug/1104069 - Don’t try to delete an empty file list. * Merged in lp:~scowcron/duplicity/ftp_password_pexpect - Use common backend.Backend get_password() rather than _ssh_pexpect.py specific code. * Merged in lp:~mhu-s/duplicity/swiftbackend - This branch adds support for Swift, the OpenStack Object Storage service. See https://blueprints.launchpad.net/duplicity/+spec/swiftbackend * Merged in lp:~verb/duplicity/boto-gcs - These patches add support for Google Cloud Storage via the boto backend. - boto has supported GCS in interoperability mode for a few years now. This change adds support by taking advantage of boto’s storage_uri abstraction layer. * Merged in lp:~ckornacker/duplicity/megacloud - Add support for Mega (mega.co.nz) backend. * Applied patch from Eric S Raymond to man page to fix markup problems. * Merged in lp:~ed.so/duplicity/man.page - update paramiko links - add command parameters to synopsis - add –compare-data - some polishing and several improvements
Enhancements: * Merged in lp:~ed.so/duplicity/24syntaxfix - fix python 2.4 vs 2.5 syntax error * Merged in lp:~mterry/duplicity/u1-oauthlib - As the Ubuntu packager for duplicity, I would prefer u1backend.py used oauthlib instead of oauth. oauthlib is well maintained upstream (unlike oauth), has a python3 port (for the future), and is in Ubuntu main (so is oauth right now, but hopefully in the future we can drop it to universe, in which case duplicity can’t use it anymore). * Merged in lp:~mterry/duplicity/delete-new-sig-in-cache - In duplicity 0.6.20, we fixed bug 1031269. This means that we no longer leave sig files on the remote location. Leaving sig files on the remote location also caused a bug with deleting cache files. Code used to leave remote new-sig but delete the locale cache new-sig; this meant that we would keep downloadoing the new-sig all the time from remote. We had worked around that by just not deleting the new-sig in the cache, which was sort of the wrong side of that problem to tackle. Now that we handle the remote new-sigs better (by deleting them), I don’t think we need this code anymore. Patch by email@example.com. * Merged in lp:~mterry/duplicity/u1-ascii-error - Fix for u1backend unicode error. Patch by Paul Barker. * Merged in lp:~satwell/duplicity/caching - Add a cache for password and group lookups. This significantly improves runtime with very large password and group configurations. * Merged in lp:~ed.so/duplicity/manpage - more formatting fixes, clarifications in sections EXAMPLES, FILE SELECTION * Merged in lp:~ed.so/duplicity/lftp.netrc - Allow .netrc auth for lftp backend * Merged in lp:~mterry/duplicity/946988 - This fixes bug 946988 by not duplicating the checks for when we should ask for the password (those same checks are done more correctly inside get_passphrase). And add a test to reproduce the bug. * Merged in lp:~lenharo-h/duplicity/duplicity - Generate encrypted backups without revealing the user’s key id via option –hidden-encrypt-key * Merged in lp:~mterry/duplicity/u1-utf8 - Make sure u1backend returns filenames as utf8 * Merged in lp:~carlos-abalde/duplicity/gdocs-backend-gdata-2.0.16.-upgrade - Upgrade of GoogleDocs backend to python gdata lib >= 2.0.15: Stop using get_everything method. * Merged in lp:~ed.so/duplicity/webdav.fix-retry - bugfix: webdav retrying broke on ERRORS like “error: [Errno 32] Broken pipe” in socket.pyas reported here https://answers.launchpad.net/duplicity/+question/212966 added a more generalized ‘retry_fatal’ decorator which makes retrying backend methods even easier * Merged in lp:~ed.so/duplicity/manpage - Clear up PASSPHRASE reusage as sign passphrase. Minor fixes. * Merged in lp:~ed.so/duplicity/u1_and_manpage - Manpage - document Ubuntu One required python libs - added continuous contributors and backend author notes - U1backend - lazily import non standard python libs, fixes http://article.gmane.org/gmane.comp.sysutils.backup.duplicity.general/5753 - fix “not bytearray” prevents PUT with python 2.6 - don’t hang after putting in credentials (cause it silently retries in background) but go through with backup * Fixed 1091269 Data corruption when resuming with –no-encryption - Patches from Pascual Abellan that make block size consistent and that add -n (no-encryption) option to manual-ctrl-c-test.sh. - Modified gpg.py patch to use 64k block size so unit test passes. * Merged in lp:~mterry/duplicity/static-corruption - This branch fixes three possible ways a backup could get data-corrupted. Inspired by bug 1091269. A) If resuming after a volume that ended in a one-block file, we would skip the first block of the next file. B) If resuming after a volume that ended in a multi-block file, we would skip the first block of the next file. C) If resuming after a volume that spanned a multi-block file, we would skip some data inside the file. - A and B are because when finding the right place in the source files to restart the backup, the iteration loop didn’t handle None block numbers very well (which are used to indicate the end of a file). - C is what bug 1091269 talks about. This was because data block sizes would get smaller as the difftar file got closer and closer to the volsize. Standard block sizes were 64 * 1024. But say we were close to the end of the difftar… When resuming, duplicity doesn’t know the custom block sizes used by the previous run, so it uses standard block sizes. And it doesn’t always match up, as you can imagine. So we would leave chunks of data out of the backed up file. - Tests added for these cases. - This branch is called ‘static-corruption’ because all these issues occur even when the source data doesn’t change. I still think there are some corruption issues when a file changes in between duplicity runs. I haven’t started looking into that yet, but that’s next on my list. - C only happened without encryption (because the gpg writer function already happened to force a constant data block size). A and B happened with or without encryption. * Merged in lp:~ed.so/duplicity/webdav.fix-retry - added ssl certificate verification (see man page) - more robust retry routine to survive ssl errors, broken pipe errors - added http redirect support * Merged in lp:~ed.so/duplicity/webdav.manpage - explanation of webdav changes above * Merged in lp:~mterry/duplicity/pygi - Python bindings for the gobject stack (used in the gio backend) have changed from static to dynamically-generated bindings. The old static bindings are deprecated. So here’s a branch to change the gio backend from old to new ones. * Merged in lp:~mterry/duplicity/py3rsync - This branch lets one build the _librsync module with Python 3. You can’t really do anything useful with it, but it’s a nicely-isolated piece to add Python 3 support for. - The changes are a mix of modernization and #ifdef logic. - All tests still pass in Python 2.7 and 2.4. I tested manually that the module worked as expected in Python 3. * Merged in lp:~duplicity-team/duplicity/po-updates - Updated translations
Enhancements: * Merged in lp:~ed.so/duplicity/ssh.manpage - added gdocs and rsync REQUIREMENTS - added cloudfiles documentation * Merged in lp:~ed.so/duplicity/gpginterface - refactor GnuPGInterface to gpginterface.py reasoning can be found in README * Merged in lp:~ed.so/duplicity/duplicity.helpfix - fix rare ‘TypeError: encode() argument 1 must be string, not None’ - http://lists.nongnu.org/archive/html/duplicity-talk/2012-09/msg00016.html * Merged in lp:~ed.so/duplicity/duplicity.tmpspacefix - use tempfile.TemporaryFile() so unused temp files are deleted automagically - propbably solve bug ‘Out of space error while restoring a file’ - https://bugs.launchpad.net/duplicity/+bug/1005901 - http://lists.gnu.org/archive/html/duplicity-talk/2012-09/msg00000.html * Merged in lp:~mterry/duplicity/utf8-po - For some crazy reason, the gettext module defaults to giving you strings in whatever charset the po file happened to define. Which means you never know what string of bytes you’re going to get. This module makes sure we always get utf-8 byte strings. So we’re at least predictable and reduces one source of UnicodeDecodeErrors (like in bug 989496) * Merged in lp:~mterry/duplicity/1031277 - ssh: actually delete all the requested files, not just the first one * Merged in lp:~mterry/duplicity/leftover-sigtar - So currently, duplicity does not delete signature files when doing a remove-all-but-n operation. Seems wrong, since those signature files are now useless and take up space. - This branch does several things: 1) Make remove-all-but-n operate on chains. In practice it did before, since the sets it operated on always came from complete chains (i.e. it never used only some of the sets from a chain) 2) Add a new method to get all signature chains before a certain time. 3) Use this new method to also delete signature chains during remove-all-but operations. - And it cleans up the cleanuptest.py file: 1) Removes crufty, unused code 2) Disallows changing the destination folder for the test, which no one would ever want to do and isn’t really supported anyway 3) Add some additional checks to the existing test 4) Adds two new methods to test remove-all-but-n and remove-all-inc-of-but-n-full * Merged in lp:~ed.so/duplicity/duplicity.manpage - disabled hyphenation and block justification for better readablility of command line examples. - reformatted REQUIREMENTS section for hopefully better online rendering - minor clarifications * Merged in lp:~gregretkowski/duplicity/cf-retry-delete - This will retry cloudfile delete commands. With large numbers of archive files over mediocre links transient network errors will occasionally cause deletes to fail and these should be retried. * Merged in lp:~mterry/duplicity/ropath.index - This branch does two main things: 1) Skips base dir entries when compiling the list of deleted delta iters. (this gracefully recovers from the sort of situations that lead to bug 929067). I’m reasonably confident this is an uninvasive change, but please confirm. 2) Overwrites the sigtar file on backup-restart. This is because AFAICT, duplicity will rewrite the entire sigtar each restart. But we were opening the sigtar file as “ab”, so we’d just dump the contents on top of the previous contents. Which was causing some confusion in bug 929067. If I’m wrong that we don’t always rewrite the entire sigtar each time, this needs some rethink. Please also confirm that. - In addition, I added two tests for the above two changes and make some improvements elsewhere in the restarttest.py file while I was at it. * Merged in lp:~ed.so/duplicity/ssh-pexpect-msgbug - Fixes ‘UnboundLocalError: local variable ’msg’ referenced before assignment’ in _ssh_pexpect.py * Merged in lp:~ed.so/duplicity/gpg.tmp - place gpg.py tempfiles in duplicity’s tmp subfolder which is cleaned whatever happens * Merged in lp:~mterry/duplicity/u1-402 - Switch the code we check for out-of-space in u1backend. * Applied patch for #1066625 ubuntu one backend - add delay between retries
Bugs closed in this release: 519948 remove-* commands don’t remove signature-files 907077 assert len(chain_list) == 2 AssertionError 929067 crash during restore: “assert ropath.index == (), ropath.index” 995851 doc improvement for –encrypt-key, –sign-key 997691 Ubuntu One backend should check for 402 error for out-of-space 1031269 remove-all-but-n-full broken 1039001 –exclude-if-present and –exclude-other-filesystems causes crash with inaccessible other fs 1066625 ubuntu one backend does not work without gnome/dbus/x11 session
Enhancements: * lots of work on the man page to clean up requirements, etc. * use empty listbody for enhanced webdav compatibility * initial folder creation on backend does not result in a ResponseNotReady anymore * add ssh_config support (/etc/ssh/ssh_config + ~/.ssh/config) to paramiko sshbackend * add missing_host_key prompt to new sshbackend similar to ssh procedure * added –ssh-backend parameter to switch between paramiko,pexpect * allow answering gio mount questions (albeit naively) * if the gio backend wants to ask a question during its mount phase, it previously just aborted. * a couple more warning error codes that Deja Dup is interested in noticing. * ssh paramiko backend respects –num-retries now * set retry delay for ssh backends to 10s * ssh pexpect backend + sftp part does not claim ‘Invalid SSH password’ although it’s only ‘Permission denied’ now + sftp errors are now more talkative * gpg.py + commented assert which broke otherwise working verify run
Bugs closed in this release: 588541 Connection failed, please check your password: Login dialog cancelled
Merges: lp:ed.so/duplicity/0.6-manpage lp:ed.so/duplicity/0.6-webdav_fixes lp:carlos-abalde/duplicity/gdocs-backend-gdata-2.0.16.-upgrade. lp:ed.so/duplicity/0.6-ssh_add_missinghostkey lp:~ed.so/duplicity/0.6-readd_sshpexpect
Enhancements: * fix extraneous ‘.py’ in botobackend.py include * tests: add delay between backups to avoid assertion error * tests: use backup source that is more likely to be larger than 1M compressed * tests: make other-filesystem check more robust against certain directories being mounts or not * resuming an incremental results in a ‘Restarting backup, but current encryption settings do not match original settings’ error because curtime is incorrectly set away from previous incremental value * added option to not compress the backup, when no encryption is selected * always delay a little bit when a backend gives us errors * Don’t cache TarInfo files. Tests still pass, so I don’t believe we need the members cache (and in the old tarfile.py, we didn’t cache either). * Adding –file-prefix option so different sets of backups can be stored in the same bucket. See blueprint at https://blueprints.launchpad.net/duplicity/+spec/file-prefix-option * two changes that help the test suite pass * raise log level on backend import failure so it will be visible under default conditions * file /etc/motd may not exist in test environment. Use file instead to point to a known plaintext source file. * some code/import changes to make the ssh and boto backends compatible with Python 2.4. * some changes to make roottest.py compatible with the new dir structure.
Bugs closed in this release: 884638 Python 2.5 / boto error 908228 possible memory leak 909031 SSH-Backend: Creating dirs separately causes a permissons-problems 916689 multipart upload fails on python 2.7.2 929465 UnsupportedBackendScheme: scheme not supported in url: scp://firstname.lastname@example.org/foo/ 930727 ftpsbackend should respect num_retries for ftp commands 931175 duplicity crashes when PYTHONOPTIMIZE is set
Merges: lp:mterry/duplicity/always-delay lp:mterry/duplicity/memleak lp:mterry/duplicity/nopexpect lp:mterry/duplicity/resume-inc lp:mterry/duplicity/testfixes lp:nguyenqmai/duplicity/file-prefix-option lp:~tobias-genannt/duplicity/nocompress
Enhancements: * Added –rsync-options flag to allow user to pass options to rsync at will * Added –s3-use-multiprocessing to select the new s3 multiprocessing backend. Default is to use the single processing backend. A helper, filechunkio.py, requires Python 2.6+, so this option is not usable in earlier versions.
Bugs closed in this release: 411145 Misleading error message: “Invalid SSH password” 871875 File … was corrupted during upload. 878220 UnsupportedBackendScheme: scheme not supported in url: s3+http 878964 Resuming a backup with a different password should throw an error 881070 Bugfix for S3 multipart upload on 0.6.16 881727 duplicity 0.6.16 lists all backup sets as incomplete 885670 Duplicity fails to add incremental backups to chain…
Merges: lp:mterry/duplicity/check-passphrase-on-restart lp:moss-d/duplicity/rsync-options
Enhancements: * Usability enhancement: sign passphrase prompt has no second verification prompt anymore, symmetric passphrases are still verified * Fixed Unicode errors when translations are used. * Replaced old tarfile.py with Python 2.7 version, modded to support Python 2.4 syntax.
Bugs closed in this release: 485219 Fixed fields are not fixed, leading to buffer overflows… 676109 Amazon S3 backend multipart upload support 690549 uid and guid setting adversely affects integrity 739438 Local backend should always try renaming instead of copying 832149 Uploads to Rackspace fail silently 835892 duplicity crash: “AssertionError: rb None None” 838162 Duplicity URL Parser is not parsing IPv6 properly 838264 Duplicity thinks partial encrypted backups are not encrypted 870116 Duplicity does not handle UIDs higher than 60001
Merges: lp:ed.so/duplicity/reuse-passphrase-for-signing-fix lp:mterry/duplicity/cloudfiles-10k lp:duplicity-team/duplicity/check-volumes lp:mterry/duplicity/tarfile lp:mterry/duplicity/partial-encryption lp:mterry/duplicity/fix-local-backend-validation lp:ross-ross-williams/duplicity/gpg-agent-fix lp:mterry/duplicity/rbNoneNone lp:~ed.so/duplicity/UnicodeDecodeError
Enhancements: * Ignore 404 errors when deleting a file on Ubuntu One. * Ignore ENOENT (file missing) errors where it is safe. * Set minimum Python version to 2.4 in README. * introduce –numeric-owner parameter patch courtesy of Lukas Anzinger <l.anzinger AT gmail.com> * duplicity:restore_check_hash “Invalid data - *** hash mismatch” lists the offending filename * fixes to unit tests to support SIGN_PASSPHRASE
Bugs closed in this release: 524922 duplicity does not have numeric uid/gid support 703142 AssertionError: assert len(chain_list) == 2 794576 Transport endpoint is not connected 815635 Bad passphrase can leave bogus sigtar in archive 818178 Shouldn’t try to delete files it knows don’t exist 821368 Error doing backup of the .evolution folder 823556 sftp errors after rev 740 change 824678 0.6.14 Fails to install on 8.04 LTS (Hardy)
Merges: lp:mterry/duplicity/u1-ignore-404 lp:mterry/duplicity/guard-tarinfo lp:mterry/duplicity/enotconn lp:mterry/duplicity/look-at-partials-during-sync lp:mterry/duplicity/more-accurate-sync lp:mterry/duplicity/report-encrypted-chains lp:mterry/duplicity/815635 lp:mterry/duplicity/retry-u1 lp:mterry/duplicity/818178 lp:ed.so/duplicity/encr-sign-key2 lp:mterry/duplicity/u1-fixes lp:carlos-abalde/duplicity/google-docs lp:~ed.so/duplicity/numowner+hashverbose
Enhancements: 452342 Provide Ubuntu One integration
Bugs closed in this release: 433591 AttributeError: FileobjHooked instance has no attribute ‘name’ 487720 Restore fails with “Invalid data - SHA1 hash mismatch” 507904 Cygwin: Backup fails with “IOError: [Errno 13] Permission denied” 512628 –exclude-filelist-stdin and gpg error with/without PASSPHRASE 680425 Endless retype passphrase when typo 705499 “include-filelist-stdin” not implemented on version 0.6.11 739438 [PATCH] Local backend should always try renaming instead of copying 753858 cannot import name S3ResponseError 761688 Difference found: File X has permissions 666, expected 666 777377 collection-status asking for passphrase 778215 ncftpls file delete fails in ftpbackend.py 782294 create tomporary files with sftp 782321 duplicity sftp backend should ignore removing a file not there 792704 Webdav(s) url scheme lacks port support 782294 create tomporary files with sftp 782337 sftp backend cannot create new subdirs on new backup 794123 Timeout on sftp command ‘ls -1’ 797758 Duplicity ignores some FatalErrors 793096 Allow to pass different passwords for –sign-key and –encrypt-key
Merges: lp:ed.so/duplicity/0.6-add_sftp lp:ed.so/duplicity/0.6-nonfatal-exclude lp:lekensteyn/duplicity/multipass lp:mterry/duplicity/797758 lp:mterry/duplicity/gio-name lp:mterry/duplicity/levelName lp:mterry/duplicity/retry-decorator lp:mterry/duplicity/u1-status
Enhancements added this release: New manual test to make Ctrl-C issues easier to replicate. Use python-virtualenv to make testing multiple Python versions easier. In boto backend check for existing bucket before trying to create.
Bugs closed in this release: 579958 Assertion error “time not moving forward at appropriate pace” 613244 silent data corruption with checkpoint/restore 731905 File “/usr/bin/duplicity”, error after upgrade from 6.11 to 6.12
Enhancements added this release: 626915 ftps support using lftp (ftpsbackend)
Bugs closed in this release: 486489 Only full backups done on webdav 578663 Use log codes for common backend errors 581054 Inverted “Current directory” “Previous directory” in error message 620163 OSError: [Errno 2] No such file or directory 629136 sslerror: The read operation timed out with cf 629984 boto backend uses Python 2.5 conditional 655797 symbolic link ownership not preserved 670891 Cygwin: TypeError: basis_file must be a (true) file … 681980 Duplicity 0.6.11 aborts if RSYNC_RSH not set 700390 Backup fails silently when target is full (sftp, verbosity=4) 704314 Exception in log module
Merges: lp:mterry/duplicity/backend-log-codes3 lp:blueyed/duplicity/path-enodev-bugfix
Bugs closed in this release: 433970 Add an option to connect to S3 with regular HTTP (and not HTTPS) 578663 Use log codes for common backend errors 631275 missing ssh on rsyncd url - rsync: Failed to exec ssh: … 635494 backed up to S3, wiped drive, reinstalled, unable to restore backup 637556 os.execve should get passed program as first argument 669225 sftp: “Couldn’t delete file: Failure’” only logged on level 9 655468 0.6.10 does not work with S3 674506 RsyncBackend instance has no attribute ‘subprocess_popen_persist’
Merges: lp:blueyed/duplicity/bug-669225 lp:duplicity-team/duplicity/po-updates lp:ed.so/duplicity/0.6.10-backend_fixes lp:ed.so/duplicity/RSYNC_RSH-fix lp:ed.so/duplicity/sign_symmetric2 lp:ed.so/duplicity/survive_spaces lp:l2g/duplicity/use-py.test lp:mbp/duplicity/433970-non-ssl
Bugs closed in this release: 542482 Offer command to remove old incremental backups from target 578663 Use log codes for common backend errors 589495 duplicity –short-filenames crashes with TypeError 612714 NameError: global name ‘parsed_url’ is not defined 613448 ftpbackend fails if target directory doesn’t exist 615449 Command-line verbosity parsing crash
Bugs closed in this release: 502609 Unknown error while uploading duplicity-full-signatures 539393 Duplicity returns 1 when continuing an interrupted backup 550455 duplicity doesn’t handle with large files well 567738 –ssh-options options passing options to ssh do not work 576564 username not url decoded in backend (at least rsync) 579958 Assertion error “time not moving forward at appropriate pace” 582962 Diminishing performance on large files
Upgraded tahoebackend to new parse_url. Fix to warning message in sshbackend.
Fix bug where encrypted backup without –gpg-options crashes. This was a followup issue to bug 490619 released in 0.6.07. This is attempt #2 – not sure what happened to the patch, but it did not show up in 0.6-series like it should have.
Fix bug where encrypted backup without –gpg-options crashes. This was a followup issue to bug 490619 released in 0.6.07.
Bugs closed in this release: 519110 Need accurate man page info on use of scp/sftp usage. 532051 rdiffdir attempts to reference undefined vars with some command args 529869 TypeError: unsupported operand type(s) for -: ‘NoneType’ and ‘int’ 530910 TypeError: unsupported operand type(s) for +: ‘NoneType’ and ‘str’
Bugs closed in this release: 459511 –tempdir option doesn’t override TMPDIR 467391 [PATCH] WebDAV backend doesn’t work 487686 re-add scp backend and make available via command line option 490619 Use optparse not getopt 497243 0.6.06, archive dir: cache desynchronization caused by remove* 501093 SSHBackend doesn’t handle spaces in path 505739 “sslerror: The read operation timed out” with S3 520470 Don’t Warn when there’s old backup to delete 522544 OSError: [Errno 40] Too many levels of symbolic links 388673 Allow renaming paths as they are restored
Merged in lp:~mterry/duplicity/list-old-chains List/keep old signature chains
Applied patches from Kasper Brand that fixed device file handling. http://lists.gnu.org/archive/html/duplicity-talk/2009-09/msg00001.html
Merged in lp:~l2g/duplicity/flag-transl-comments which cleared up how translation comments should be passed to the translators cleanly now.
Applied 422477; [PATCH] IMAP Backend Error in delete()
Merged in lp:~mterry/duplicity/iterate-warnings Add machine codes to various warnings when iterating over source files
Fix problems with unittests under Jaunty. It appears that redirection in os.system() has changed for the worse, so a workaround for now.
Fix problem in restart where there were no manifest entries and no remote volumes stored. We clean out the partial and restart.
Fixed 435975 gpg asks for password in 0.6.05, but not in 0.5.18
Merged in lp:~l2g/duplicity/test-compat from Larry Gilbert which made the testing compatible across more systems. Also fixed the remaining collectionstest bug which was trying to test with no cache present.
Bugs fixed this release: 407968 GIO backend can’t restore 408059 Failure due to _logger.log failure for content with special characters: TypeError decoding Unicode not supported 409593 deja-dup (or duplicity) deletes all signatures 412667 “duplicity remove-older-than” asks for passphrase even though not required 418170 [PATCH] file names longer then 512 symbols are not supported
One major and one minor change. The “No such file or directory” error is bad enough that this should be released quickly. For those of you using encryption, this is not a problem, but for those of you that do not use encryption (–no-encryption), then this will manifest itself if the local cache gets out of sync with the remote store.
Bugs fixed this release: 405734 duplicity fails to restore files that contain a newline character 403790 Backup error: No such file or directory
Lots of small changes and some bug fixes. * Restart error handling has been smoothed out a great deal and it “does what is right” in order to keep going. * Backends are now optional, if they fail an Info message is put out to notify of the failure and why. * There was more work on translations and internationalization. Thanks to everyone!
Bugs fixed this release: 377528 –file-to-restore doesn’t work with trailing slash 394757 Backend imports should be made optional 398230 Deja-dup backup fails with message: “Unable to locate last file” 401303 0.6.2 manpage inconsistent wrt. archive-dir/name 405646 Small i18n error 405975 duplicity.gpg.gpg_failed() breaks and spews on GnuPG error 402794 duplicity public-key-only incompatible with gnupg 2.0.11
Duplicity will now remove any spurious files left in the cache from a previous run. This will keep the metadata cache in sync with the remote storage metadata.
Bugs fixed this release: 394629 Hang on first collection-status 379386 Fix ‘list-current-files’ with missing archive dir 395826 “No such file or directory” when backing up second time 394627 User-friendly archive dir print 388699 Manifest mismatch error
Fixed issues in Checkpoint/Restart: * The –name backupname" option was added to allow the user to separate one archive from another. If not specified, the default is an MD5 hash of the target URL, which should suffice for most uses.
The archive_dir (cache) is now stored in a standard location, defaulting to ~/.cache/duplicity. See http://standards.freedesktop.org/basedir-spec/latest/
duplicity will now copy needed metadata from the remote store to the local cache as needed. This means that the first use after upgraded from 0.5.x will have the metadata copied to the local archive dir in order to sync both.
cleanup will now work correctly with the archive dir and separates the local from the remote files.
Bugs fixed this release: * 388034 Unable to backup * 378940 python2-6 issue / UTF-8 charset / Ubuntu 9.04 * 379386 Fix list-current-files w/ missing archive dir * 387102 Asynchronous upload not working properly * 387218 Make scp/ssh into sftp-only backend * 388992 List of Orphaned Files Growing * 392905 NoneType object has no attribute ‘startswith’ * 393372 Error creating directory * 383412 Add InfoCodes for upload events * 383419 Add gio backend
Checkpoint/Restart capability added. Checkpoint is done at every volume written and Restart is done at start of the next volume in the set. Changes to normal operations include a permanent duplicity archive-dir at ~/.duplicity to save state.
To accomplish this, the signature and archive files in the archive-dir now have three states: 1) temporary until the first volume has been written, 2) partial until the final volume has been written and sent to remote storage, 3) permanent with the same name as always.
Assumptions are made that if a restart is needed, then all arguments are the same as before and that no files have been removed from the file system between runs.
From now on, the –archive-dir option can be used to change the location of the archive dir, but you are responsible for moving the files if you change it.
Other fixes: Unicode filenames in log messages are now OK.
Fixed problem where Cygwin was returning -1 for the hard max open file limit.
Added support for RackSpace’s CloudFiles, cf+http.
Added support for Tahoe-LAFS from the patch, patch #6743: Tahoe backend for duplicity https://savannah.nongnu.org/patch/?6743
Only half of this bug is fixed but it’s still useful. bug #21792: pipe call fails with an error OSError: [Errno 24] Too many open files https://savannah.nongnu.org/bugs/?21792
Changed from using ulimit external command to resource.getrlimit to check open files limit.
Removed one line of code left from some testing that I did that caused a crash when the target dir was empty and collection-status was requested.
Moved from using the df command to get temp space availability to Python’s os.statvfs() call. Not all df commands work the same way.
patch #6813: Making changelist easy to read https://savannah.nongnu.org/patch/?6813
patch #6814: Ignore comments in filelists https://savannah.nongnu.org/patch/?6814
bug #24825: duplicity warn on insufficient TMPDIR space availability and low max open file limits pre-backup. https://savannah.nongnu.org/bugs/?24825
bug #25594: wrong backup statistics https://savannah.nongnu.org/bugs/?25594
bug #25976: Password requested when not needed. https://savannah.nongnu.org/bugs/?25976
patch #6806: More graceful handling of old –short-filename files https://savannah.nongnu.org/patch/?6806
Added tilde and variable expansion to the source or target argument that is not a URL.
FTP backend was failing on PureFTPd when the "-x ’‘" option was removed from the second ncftpls popen, a fix that was implemented due to bug #24741. This fix does the ls in one pass by extracting the last entry on the ’ls -l’ listing.
If a file is unreadable due to access rights or other non-fatal errors, put out error message and continue rather than dying messily with a traceback.
Added tilde ‘~’ expansion and variable expansion in the options that require a filename. You can now have this “–archive-dir=~/ArchDir/$SYSNAME” if you need it. No expansion is applied to the source or target URL’s.
Fixed problem I caused, again, where sys.exit() was trapping instead of exiting. Added big note to not to do that again.
After email voting among known duplicity contributors, the decision was reached to revert to the GPL Version 2 license, so with their consensus, duplicity is now under GPL Version 2.
Revert to calling NcFTP utilities (ls, get, put) directly rather than scripting ncftp via pexpect by reverting to the 0.5.07 version of ftpbackend.py.
Changed fatal error regarding version 3.2.0 of ncftpput to warning level since it has been reported that the problem does not occur on most distributions.
Changed from log.Log with numbered log levels to log.Debug, log.Info, log.Notice, log.Warn, log.FatalError as below: 0 log.FatalError 1-2 log.Warn 3-4 log.Notice 5-8 log.Info 9 log.Debug The -vN option has not changed. Verbosity may also be one of: character [ewnid], or word [‘error’, ‘warning’, ‘notice’, ‘info’, ‘debug’]. The default is 4 (Notice). The options -v4, -vn, and -vnotice are functionally equivalent, as are the mixed-case versions, -vN, -vNotice, -vNOTICE.
Normalized include statements and tried to insure that all duplicity includes were from the duplicity module.
patch #6790: Add –exclude-if-present https://savannah.nongnu.org/patch/?6790
Add more error detection to FTP backend.
Fix backends so sleep does not occur after last retry.
Fix so BackendException does not cause traceback except when verbosity is at level 5 or higher (Info level).
Adjust log levels so some errors show up with default verbosity.
Fixed bug where an extra comma caused a traceback during a warning about unnecessary sig files. Plus fixed print so the real filename would show up and not a Python object representation.
Add Changelog.GNU to website and distribution to add a bit of detail showing the CVS changes via rcs2log. Added dist/mkGNUChangelog.sh.
bug #22908: Don’t block gpg-agent https://savannah.nongnu.org/bugs/?22908
To fix the above, –use-agent was added as a command line option. When this is specified and asymetric encryption is enabled, then all GnuPG passphrases will come from the gpg-agent or equivalent program and no passphrase prompt will be issued.
bug #25787: Usernames with escaped @-sign are not handled properly https://savannah.nongnu.org/bugs/?25787
bug #25976: Password requested when not needed. https://savannah.nongnu.org/bugs/?25976
patch #6787: import duplicity.GnuPGInterface explicitly https://savannah.nongnu.org/patch/?6787
bug #25838: Backup fails / ncftp - remote file already exists https://savannah.nongnu.org/bugs/?25838 With this fix we also get resume in ftp get/put. If a put or get fails part of the way through, ncftp will resume on the next retry.
bug #25853: duplicity fails with boto passwords coming from ~/.boto https://savannah.nongnu.org/bugs/?25853
patch #6773: Make user name optional in rsync backend https://savannah.nongnu.org/patch/?6773
GPG errors will no longer cause tracebacks, but will produce a log entry, from gpg, similar to the following: ===== Begin GnuPG log ===== gpg: BAD0BAD0: skipped: public key not found gpg: [stdin]: encryption failed: public key not found ===== End GnuPG log ===== This will let the user know what really caused the GPG process to fail, and what really caused errors like ‘broken pipe’.
Add Epydoc output to web site and start adding documentation. http://duplicity.nongnu.org/epydoc/index.html
bug #25787: Usernames with @-sign are not handled properly https://savannah.nongnu.org/bugs/?25787
Bug #333057: GnuPGInterface prints exit statuses incorrectly https://bugs.launchpad.net/bugs/333057
bug #25696: ncftp error w/0.5.09 – nested target directories https://savannah.nongnu.org/bugs/?25696
bug #15664: When restoring backup: “OverflowError: long int too large to convert to int” https://savannah.nongnu.org/bugs/?15664
patch #6761: More robust pexpect handling of SSH authentication https://savannah.nongnu.org/patch/?6761
patch #6762: Wrong exit() used for 2.3/2.4 Python https://savannah.nongnu.org/patch/?6762
The default filename format has changed from W3 style to a long numeric style, YYYYMMDDTHHMMSSZ, with no delimiters, thus is now compatible with Windows/Samba filesystems. The time is UTC, not local, so there will be no timezone or daylight savings time issues.
Duplicity still recognizes the old long filename format, and will continue incremental backup chains if found. The old format is still available via the –old-filenames option (pending deprecation).
Users of –short-filenames or –time-separator should stop using these options on their next full backup. The new filenames are compatible with your system.
The following options are pending deprecation and will be removed in a future release: –time-separator –short-filenames –old-filenames
bug #19988: Incompatibility to Samba/SMB share https://savannah.nongnu.org/bugs/?19988
bug #25097: Allow listing files from any time, not just current time https://savannah.nongnu.org/bugs/?25097
bug #25550: Error codes do not propagate from log to exit status https://savannah.nongnu.org/bugs/?25550
bug #25308: Signatures orphaned if from another time zone https://savannah.nongnu.org/bugs/?25308
Bug #229826: duplicity crashed with ValueError in port() https://bugs.launchpad.net/duplicity/+bug/229826
FTP is now driven with pexpect rather than NcFTP utilities. This closes the following bugs (and solves other problems): bug #24741: ncftpls -x ’’ causes failure on Yahoo FTP server bug #23516: duplicity/ncftpget not closing unlinked files, …
bug #25509: Logic error in imapbackend.py [IMAP_SERVER] https://savannah.nongnu.org/bugs/?25509
bug #25512: [Patch] Retry on Imap failure https://savannah.nongnu.org/bugs/?25512
bug #25530: commandline passwd not working https://savannah.nongnu.org/bugs/?25530
Turns out going backwards in the license is not as easy as forwards. Restoring GPLv3 license until consensus reached.
bug #25293: IOError: [Errno 22] Invalid argument https://savannah.nongnu.org/bugs/?25293
bug #25379: sys.exit() causes traceback and should not https://savannah.nongnu.org/bugs/?25379
bug #25403: 0.5.06 “manifests not equal, different volume numbers” https://savannah.nongnu.org/bugs/?25403
patch #6729: New imap backend. Replaces current gmail backend https://savannah.nongnu.org/patch/?6729
patch #6730: Fix timing out for SSH backend https://savannah.nongnu.org/patch/?6730
patch #6733: Improve error handling in imapbackend.py https://savannah.nongnu.org/patch/?6733
Increase default volume size (–volsize) to 25M from 5M. This reduces the number of volumes to accomodate larger backups.
Reworked patch 6701 to list collection one at a time rather than writing all as one huge list. Was causing memeory problems when the collections got large.
Fix backendtest.py so that empty URL’s in config.py cause the backend test to be skipped rather than erroring. Added notes in config.py.tmpl explaining the change.
Add/update copyright statements in all distribution source files and revert duplicity to GPL version 2 license.
Original fix to disallow use of ncftpput 3.2.0 mistyped the ErrorCode used and resulted in an error rather than an explanation.
Fix to deprecation warnings about sha and md5 modules. Uses hashlib if available, otherwise original module.
Added loop to run-all-tests.sh to run all tests against all supported versions of Python if available. Looks for 2.3, 2.4, 2.5, 2.6.
Noah Spurrier has given us permission to distribute pexpect.py along with duplicity, so this will no longer be an install requirement.
NcFTP version 3.2.0 will not work with duplicity since we require the use of both -f and -C options on ncftpput. 3.1.9, 3.2.1+ work fine. I put in error checks for this situation in the FTP backend code.
bug #25230: –include-globbing-filelist only including first entry. https://savannah.nongnu.org/bugs/?25230
bug #25239: Error during clean, wrong case in duplcicity https://savannah.nongnu.org/bugs/?25239
patch #6709: Report correct number of volumes when restoring https://savannah.nongnu.org/patch/?6709
sr #106583: document the need to use the –force option https://savannah.nongnu.org/support/?106583
bug #25194: Duplicity 5.04 requires python-distutils-extra… https://savannah.nongnu.org/bugs/?25194
patch #6678: Add progress metering https://savannah.nongnu.org/patch/?6678
patch #6686: Add error codes for all fatal errors https://savannah.nongnu.org/patch/?6686
bug #25090: Typos and trailing whitespace in duplicity manpage https://savannah.nongnu.org/bugs/?25090
bug #24889: NCFTP cannot deal with some FTP servers https://savannah.nongnu.org/bugs/?24889
patch #6692: Print collection status in a machine-readable way https://savannah.nongnu.org/patch/?6692
patch #6693: Some FatalError’s don’t have codes still https://savannah.nongnu.org/patch/?6693
patch #6694: Log exceptions https://savannah.nongnu.org/patch/?6694
patch #6695: Log filenames https://savannah.nongnu.org/patch/?6695
patch #6696: Consolidate get_delta_iter and get_delta_iter_w_sig https://savannah.nongnu.org/patch/?6696
patch #6697: Always log at least one progress during dry run https://savannah.nongnu.org/patch/?6697
patch #6700: Make duplicity translatable https://savannah.nongnu.org/patch/?6700
patch #6701: Make current-list command machine-readable https://savannah.nongnu.org/patch/?6701
patch #6702: handle unknown errnos in robust.py https://savannah.nongnu.org/patch/?6702
GPG was throwing “gpg: [don’t know]: invalid packet (ctb=14)” and apparently this is non-fatal. There is a fix for this being rolled into GPG 2.x. http://lists.gnupg.org/pipermail/gnupg-devel/2006-September/023180.html Copied from collections.py. Fix supplied by Simon Blandford email@example.com
bug #24731: Documentation error: “if… if” in remove-older-than paragraph https://savannah.nongnu.org/bugs/?24731
bug #24775: Digest Auth for WebDAV backend https://savannah.nongnu.org/bugs/?24775
patch #6676: Raw delta stats aren’t right for multivolumes https://savannah.nongnu.org/patch/?6676
patch #6675: Add modelines https://savannah.nongnu.org/patch/?6675
patch #6674: Add –log-* options to man page https://savannah.nongnu.org/patch/?6674
patch #6673: Add –dry-run option https://savannah.nongnu.org/patch/?6673
patch #6672: makedist doesn’t ship util.py https://savannah.nongnu.org/patch/?6672
patch #6670: Machine Readable Output https://savannah.nongnu.org/patch/?6670
patch #6662: improve s3 backend error reporting https://savannah.nongnu.org/patch/?6662
patch #6652: improve asynch scheduler (including the synchronous case) https://savannah.nongnu.org/patch/?6652
patch #6642: make ParsedUrl() thread-safe with respect to itself https://savannah.nongnu.org/patch/?6642
patch #6638: correct typo in reporting lack of sufficiently new boto backend https://savannah.nongnu.org/patch/?6638
sr #106496: put install-from-cvs-notes in CVS-README https://savannah.nongnu.org/support/?106496
sr #106534: GMail backups aren’t stored in the correct location https://savannah.nongnu.org/support/?106534
Add -h option for help
Change gpg logging so that logs are always collected. The log is printed in the case of gpg IO errors. Also, verbosity level 5 or above (-v5) will print the logs.
patch #6297: Add IMAP/s/gmail support https://savannah.nongnu.org/patch/?6297
bug #24260: backend.py missing re import https://savannah.nongnu.org/bugs/?24260
bug #24274: asyncscheduler.py missing sys import https://savannah.nongnu.org/bugs/?24274
bug #24234: Tabs Present In Source Files https://savannah.nongnu.org/bugs/?24234
bug #24223: WebDAV backend broken in 0.5.00 https://savannah.nongnu.org/bugs/?24223
bug #24226: WebDAV Does Not Create Collection If Needed https://savannah.nongnu.org/bugs/?24226
Changes to unit tests: - resolve circular imports after backend reorg - resolve exception error import - now in errors.py - remove need for temp2.tar to be in CVS repository
bug #23988: scp destination fails if no username is specified https://savannah.nongnu.org/bugs/?23988
bug #23985: –no-encryption option does not work in 0.4.12 https://savannah.nongnu.org/bugs/?23985
patch #6623: slightly augment tempdir cleanup logging https://savannah.nongnu.org/patch/?6623
patch #6596: re-organize backend module structure https://savannah.nongnu.org/patch/?6596
patch #6589: S3 european bucket support https://savannah.nongnu.org/patch/?6589
patch #6353: Concurrency for volume encryption and upload. https://savannah.nongnu.org/patch/?6353
Dan Muresan created a patch to minimize the number of password prompts. To do so, it sometimes requests a password once without confirmation; if later it turns out that a full backup is needed, the user is prompted for confirmation.
bug #23540: doc bug in man page (environment FTP_PASSWORD) https://savannah.nongnu.org/bugs/?23540
bug #23362: Documentation for –version, –time-separator
bug #23283: interactive passphrase query is suboptimal https://savannah.nongnu.org/bugs/?23283
bug #23066 was not actually applied to 0.4.11. Its here now. https://savannah.nongnu.org/bugs/?23066
bug #22826: regressions caused by boto 1.1c https://savannah.nongnu.org/bugs/?22826
Changes applied to allow duplicity to run under Python 2.3 again.
patch #6485: Reinstate patch #6340 with a detailed explanation. https://savannah.nongnu.org/patch/?6485
bug #23066: ssh uris with given portnumbers are not handled correctly https://savannah.nongnu.org/bugs/?23066
bug #22728: FTP backend fails on empty directory https://savannah.nongnu.org/bugs/?22728
patch #6374: Duplicity –tempdir patch documentation. https://savannah.nongnu.org/patch/?6374
patch #6375: Duplicity reports the epoch for a nonexistant last full backup date https://savannah.nongnu.org/patch/?6375
patch #6380: add additional named logging levels https://savannah.nongnu.org/patch/?6380
patch #6389: Possible Fix for pagefile.sys on Win32 systems https://savannah.nongnu.org/patch/?6389
patch #6403: Restore by overwriting files/directories by using –force option https://savannah.nongnu.org/patch/?6403
patch #6449: add additional debug level logging https://savannah.nongnu.org/patch/?6449
patch #6453: handle absolute urls in webdav backend https://savannah.nongnu.org/patch/?6453
Fix problem where S3 prefix was prepended with ‘d’. This caused a failure in the regression tests.
NOTE: URL format correction in rsync://. The rsync backend now properly supports absolute and relative pathnames and module access. The formats are: rsync://user@host::/module/some_dir rsync://user@host/relative_path rsync://user@host//absolute_path
Fixed regression caused by changeover to new urlparse.py. bug #21475: FTP Usernames that contain ‘@’ are not recognized https://savannah.nongnu.org/bugs/?21475
Added section URL FORMAT in the duplicity man page.
Added 2nd patch to bug #21475 that forces all versions of Python to use the fixed urlparse.py.
Fixed so that remove-older-than and remove-all-but-n-full will not request a GPG passphrase.
Fixed issue with Pure-FTPd that would always return an empty directory listing and thus force a full backup every time. A side effect of the change is that we now only make one call to ncftpls to get the listing, thereby reducing the overhead on systems with a large number of backup files.
bug #21896: Two problems with rsync under 0.4.8 + patch https://savannah.nongnu.org/bugs/?21896
bug #21909: Problematic typo in compare_verbose() method https://savannah.nongnu.org/bugs/?21909
patch #6354: S3 staight typo results in a bogus exception https://savannah.nongnu.org/patch/?6354
patch #6356: Command line option for the temporary directory root. https://savannah.nongnu.org/patch/?6356
patch #6357: Explicit restore action is missing from the command list, https://savannah.nongnu.org/patch/?6357
First pass at bringing unittest cases up to date. All unit tests are working, but more test cases need to be added to handle the new protocols, plus there is some print leakage even with logging turned off.
Allow pexpect to force the close of the child on sftp calls. We already do that with scp calls. This cleans up that exception.
bug #21751: rsync module urls do not work in 0.4.7 https://savannah.nongnu.org/bugs/?21751
bug #21752: Boto backend needs version 0.9d or later https://savannah.nongnu.org/bugs/?21752
patch #6340: S3 short filename regression https://savannah.nongnu.org/patch/?6340
patch #6344: S3 bad bad key key handling http://savannah.nongnu.org/patch/?6344
Applied patch from Eric Hanchrow to fix logging error in botoBackend, and fix delete() in rsyncBackend.
bug #21673: remove-all-but-n-full wrong arg usage https://savannah.nongnu.org/bugs/?21673
bug #21686: NcFTPGet 3.2.0 tempfile incompatibility https://savannah.nongnu.org/bugs/?21686
patch #6292: Amazon S3 bucket creation deferral https://savannah.nongnu.org/patch/?6292
patch #6293: left-over patch from remove-all-but-n-full https://savannah.nongnu.org/patch/?6293
patch #6296: Generic S3 url support for Duplicity 0.4.6 https://savannah.nongnu.org/patch/?6296
patch #6298: URI unquoting patch for FTP backend https://savannah.nongnu.org/patch/?6298
patch #6299: re-design tempfile handling https://savannah.nongnu.org/patch/?6299
patch #6300: Standard library replacement for ParsedUrl class https://savannah.nongnu.org/patch/?6300
patch #6301: log sftp commands at verbosity 5 https://savannah.nongnu.org/patch/?6301
https://savannah.nongnu.org/bugs/?21508 bug #21508: Change delete implementation of ftpBackend to only send one “DELE” instead of multiple per delete.
https://savannah.nongnu.org/bugs/?21646 bug #21646: –archive-dir causes delete of remote full sigs and orphaned sig files
https://savannah.nongnu.org/bugs/?21651 bug #21651, add https support for webdav.
https://savannah.nongnu.org/bugs/?21657 bug #21657: ncftpls fails to create dir in ver 0.4.5
https://savannah.nongnu.org/patch/?6284 patch #6284: document TMPDIR and friends
https://savannah.nongnu.org/patch/?6284 patch #6285: security fix: eliminate use of mktemp()
https://savannah.nongnu.org/patch/?6289 patch #6289: Amazon S3 key prefix patch for Duplicity 0.4.5
https://savannah.nongnu.org/patch/?6291 patch #6291: Alternative WebDAV HTTPS patch
https://savannah.nongnu.org/bugs/?21646 Fix to handling of collections when –archive-dir is used. Prior to this, duplicity would write the full sig files to both local and remote, then delete the remote. Now, it does not delete the remote full sigs.
Applied the following patches from Peter Schuller patch #6279, add command ‘remove-all-but-n-full’ patch #6280, clarify –archive-dir option patch #6281, –help should print to stdout, not stderr patch #6282, collection-status: output in more consistent order
All the changes in RC1 through RC4 plus:
Changes to ftpBackend to use a temp login config file rather than putting the username and password on the command line. This requires the use of NcFTP 3.1.9 or later.
Thanks to a patch from Greg Hewgill the Amazon S3 backend now uses –num-retries to retry IO repeatedly if needed.
Changes to commandline processing to allow non-ambiguous short strings for commands, i.e. ‘i’, ‘inc’, ‘incr’ for ‘incremental’, ‘f’ for ‘full’, etc.. A warning message is printed if the short command is not unique. Note: this already works for options, so I just applied the same idea to commands.
Applied a patch from Gregory Hartman to correct handling of DST in time calculations. This affects backups made the night of a DST time switch.
WARNING: COMMAND LINE CHANGES ARE NOT BACKWARDS COMPATIBLE! There is a new command line syntax to separate actions and options. Refer to the new man page for full details of the change. The new syntax looks like: duplicity [full|incr] [options] source_dir target_url duplicity [restore] [options] source_url target_dir duplicity verify [options] source_url target_dir duplicity collection-status [options] target_url duplicity list-current-files [options] target_url duplicity cleanup [options] target_url duplicity remove-older-than time [options] target_url
Fixed issue in –time-separator where the current time string was being set prior to setting the separator, causing errors when trying to set the –time-separator for Windows systems.
Fix so that file mtime is always compared in full seconds.
Fix so that ftpBackend.delete() does not print file list.
Patch from Olivier Croquette to add –full-if-older-than=
Patch from Stefan Hoth to add :port option in FTP.
Patch from Mitchell Garnaat to get all keys from S3, rather than just the first 1000.
Fix to sshBackend to version check for python-pexpect 2.1.
Fix one case in ftpBackend where host string was used instead of url_string. This only affected the creation of the target dir on the remote system, if it did not exist, and only if the user or port needed to be specified.
Add patch from Alexander Zangerl to suppress the GPG passphrase prompt when a passphrase is not needed. - full and pubkey enc: doesn’t depend on old encrypted info - inc and pubkey enc and archive-dir: need manifest and sigs, which the archive dir contains unencrypted - with encryption disabled - listing files: needs manifest, but the archive dir has that - collection status: only looks at a repository
Add patch from Olivier Croquette to allow user@domain usernames, making ftp://user@firstname.lastname@example.org/path a valid URL.
Added a bit of debug print to sshBackend for –verbosity=9.
Changed usage message to separate options and commands.
Patches applied from Debian DP: fix #228388: old/aborted/offending sig files prohibit further action DP: fixes manual page and usage msg for rsync url and –remove-older-than DP: make tempfiles with useful names DP: do not ask for a passphrase if none is required!
https://savannah.nongnu.org/bugs/?21123 duplicity 0.4.3 ftpBackend did not find backup sets when there was more than 20 files in directory.
https://savannah.nongnu.org/patch/?6212 Large performance boost for large volume sizes.
https://savannah.nongnu.org/patch/?6211 Restore strict host checking in sshBackend.
https://savannah.nongnu.org/patch/?6205 Add option –librsync-dir for when its not found.
All the changes in RC1 through RC12 plus:
Move get_password() to Backend class to standardize.
Fix problem with ftpBackend to create target directory if needed. Note: this creates only one level.
Dropped ssh-command and added ssh-options to allow users to add options to the scp and sftp commmands.
Removed use of tempfile.TemporaryFile(). This fixes the restore problem on Windows that was due to Python bug 1776696 reported on Sourceforge.
Fixed Debian Bug#437694: Make bzip2 compression optional. The default is not to do bzip2 compression. To use bzip2 add the following command line option: –gpg-options=‘–compress-algo=bzip2 –bzip2-compress-level=9’ Note: do not add spaces in the string value.
Fixed bug 20764 - unable to use port in ssh backend. https://savannah.nongnu.org/bugs/?20764
Remove ssh_command option, add ssh_options. This adds options to the scp and sftp commands that are used by the ssh backend.
Change ssh backend to send ‘quit’ instead of EOF when using sftp. This allows it to run under cron as long as the password is supplied non-interactively.
Change ssh backend to not pass :port part of URL to scp or sftp. We already supply -oPort=xx for port.
Fix index out of range in Bug 20730, triggered when there is only one incremental in the list. https://savannah.nongnu.org/bugs/?20730
Changed the file:, ftp:, and ssh: backends so that the target directory will be created at start.
Changed the ftp: backend so that empty target dirs do not error out.
Duplicity is now covered under GPL version 3 (or later).
Duplicity now correctly processes scp URL’s of the form: scp://user@host[:port]/ where the directory spec is empty. This fixes a bug where the user could not write into the home directory on the target.
The SSH/SCP backend has had an overhaul. It now requires the python-pexpect module. Normally this can be obtained from your distro’s repository, but if you want, you can download pexpect from http://pexpect.sourceforge.net.
The SSH/SCP backend work was done to allow the user to use password authentication rather than public-key. You may now enter a password, either through the FTP_PASSWORD environment variable, or at the console. To activate this feature you will need to use the option –ssh-askpass on the command line. The default is public-key, which does not look for a password from either source.
Various fixes to the man page for –ssh-askpass, –remove-older-than, and –archive-dir.
Add support for: –ftp-passive, –ftp-regular
Removed -m option on FTP put command. This means that the remote directory must exist prior to backup.
Changed ftpBackend from -f option back to commandline. Various versions of ncftp* interact differently when both -f and commandline options are supplied.
The FTP password is munged in all log operations.
Added logging of filenames in the bucket when -v9 is used on Amazon S3.
Change to a max block size of 2048 bytes for rsync difference buffer. This may slow things down for truly large files, but will give much smaller deltas on files with numerous small changes, such as database files.
New S3 backend, Boto, from Eric Evans, replaces bitBucket. Boto can be obtained from http://code.google.com/p/boto/. I did not make this a requirement for setup since its not in the normal repositories.
New FTP backend from Thorsten Schnebeck that uses ncftp instead of Pythons ftplib. This seems to be much more solid. I added the -f option with a secure temp file to contain host, user, and password, rather than having them on the command line. I also added the -m option to the put command to create the target directory and the -t option to make sure it times out if there is a network problem.
The Backend class now contains a popen_persist function that acts like run_command_persist. Both use the new num_retries global.
Added a commandline option, ‘–num-retries=
Bug 20282 - Thomas Tuttle: An out of range index when checking past history in the backup sets caused a failure when trying to access later.
Bug 20149 - dAniel hAhler: dAniel submitted a second patch for this for further cleanup. The new patch prefers the latest intact backup set.
Bug 20039 - Andreas Schildbach: –and– Patch 6030 - Alexander Zangerl email@example.com: Duplicity now uses bzip2 for compression. This matches the way the Debian distribution handles it. I’ll think about adding an option to override later, if its needed.
Bug 20179 - dAniel hAhler: When errors cause login to fail in FTP, reset and try again.
Patch 6015 -dAniel hAhler: Better display of traceback when ftpBackend errors out.
Patch 6029 - Alexander Zangerl firstname.lastname@example.org: http://bugs.debian.org/370206 archive-dir together with incremental backup results in crash. the patch is simple, the code in 0.4.2 did attempt to access strings as objects.
Patch 6031 - Alexander Zangerl email@example.com: http://bugs.debian.org/369971 there’s some problems with unattended encrypted dumps, if the user doesn’t want to hand duplicity the gpg passphrase and attempts to work around this by using a local archive dir. the patch makes it look at a manifest in a local archive dir if gpg doesn’t manage to decrypt a remote one (no surprise without a passphrase).
Patch 6032 - Alexander Zangerl firstname.lastname@example.org: a new feature patch: i’ve recently gotten annoyed with having gazillions of 5mb files and therefore added a –volsize option to allow the user setting the chunk size. the patch is simple and contains a manpage update as well.
Patch 6033 - Alexander Zangerl email@example.com: let’s add a –help terse usage message and don’t just direct the user to the manual. this should come handy if somebody needs to restore stuff without having the manual available.
Bug 20149 - dAniel hAhler: When errors cause an incomplete backup set, flag the error with a message, rather than erroring out. The user then knows to run –cleanup.
Patch 5998 - Kuang-che Wu: Cache uid and gid lookup to speed operations.
Patch 5993 - firstname.lastname@example.org: Make Amazon S3 backend incrementally more robust for recovery.
GnuPG fails when trying to access stdin on an empty passphrase. Changes allow empty passphrase on public-key encryption and now respond gracefully on empty passphrase for symmetric encryption.
dAniel hAhler submitted a patch to change “Error initializing file foo” (log level 2), where foo was a socket, to “Skipping socket foo” (log level 7). https://savannah.nongnu.org/patch/?5985
Change logging to flush after every write, unbuffering stdout and stderr, thus producing logs that are coherent.
More fixes on FTP. dAniel hAhler supplied a new patch for FTP that cleans up the error handling and reduces the retry time to zero on the first retry.
Fixed connection problem in FTP where it was not quitting on connection reset and just logging in again. This created many stale logins on the remote system.
Changed attribution of a couple of patches to dAniel hAhler, who actually wrote the patches, not just found them.
Fixed bug in tarfile.py that was causing ValueError exception. Thanks to dAniel hAhler for the patch that fixed the problem. Refer to: http://savannah.nongnu.org/bugs/?19998
Applied patches: https://savannah.nongnu.org/patch/?4486 https://savannah.nongnu.org/patch/?5183 https://savannah.nongnu.org/patch/?5185 https://savannah.nongnu.org/patch/?5412 https://savannah.nongnu.org/patch/?5413 https://savannah.nongnu.org/patch/?5680 https://savannah.nongnu.org/patch/?5681 https://savannah.nongnu.org/patch/?5682 https://savannah.nongnu.org/patch/?5794 https://savannah.nongnu.org/patch/?5830
Fixed bugs: https://savannah.nongnu.org/bugs/?2441 https://savannah.nongnu.org/bugs/?16711
ProFTPD resets the connection after returning 226 when NLSTing an empty directory, so changed code to allow that exception.
ftpBackend now asks for a password if FTP_PASSWORD does not exist.
rsyncBackend was using full URL in the commandline and failing. It now uses only server:path/, leaving off rsync://.
Added –sftp-command option, now that the scp backend uses sftp for listing and deleting files.
Brian Sutherland has contributed a new backend for Amazon’s S3 data storage service.
Added some patches compiled by Andre Beckedorf:
Tolerate more errors when listing an ftp directory (errors indicate an empty dir). Retry ftp commands when upon temporary error. (Thanks to to Stefan Schimanski and dAniel hAhler for their patches.)
Mathias de Riese’s substantial patch or patches appear to have been applied to CVS years ago, but not released yet. Sorry about that.
Followed suggestion by David Rigel, make user type passphrase twice to confirm.
Eric Hanchrow’s patch makes sure duplicity deletes older signatures when using –remove-older-than.
Jiri Tyr’s patch may fix some scp/sftp problems.
asdf’s patch makes sure uids or gids over 2097151 don’t corrupt the tarfiles.
Cleaned up and documented –collection-status option, which lists the backup chains and sets found in the repository.
FTP error 450 when listing a directory now understood to mean the directory is empty (duplicity will not exit with error).
–remove-older-than now cannot delete the active backup chain, even if you specify a time later than the chain (e.g. “–remove-older-than now”).
Applied (version of) Helmut Schneider’s patch to display file mod times with –list-current-files.
Fixed bug found by Rob Browning handling symlinks with long names that have long names.
Applied Stephen Isard’s patch to fix exclude-globbing-filelist.
Applied Sebastian Wilhelmi’s patches to add rsync as a backend.
To improve large file performance, signature block size is now based on file length.
duplicity should now build with librsync 0.9.6. Much thanks to Donovan Baarda for his work in this version of librsync.
duplicity should work with Python 2.3 now.
Changed restore procedure to download volumes from all backup sets simultaneously.
Changed the verbosity level of some messages to 5, so level 4 is cleaner.
Added –verify option, for checking whether a backup is up-to-date, or to see what has changed since that backup.
GPG no longer needed: with –no-encryption option duplicity will write gzipped volumes. The checksums of volumes will still be verified, but this will not stop a malicious attack because the manifest files can be easily updated.
Earlier versions could crash when doing an incremental backup where no files had changed.
Now by default the most recent files get restored, not the oldest. You can get the old weird behavior by specifying –restore-time 1 (or any other very early time).
Fixed a couple bugs where duplicity would crash if it found the remote directory corrupted in various ways (for instance, if it contained a file that looked like a duplicity file but had an invalid time string).
Added –cleanup option, for deleting the files that may accumulate if a duplicity session is aborted after it has uploaded some files.
Added –remove-older-than option, for safely deleting backup sets older than the given time.
Suppress GPG log messages like “gpg: CAST5 encrypted data” if verbosity is set to 3 or less.
short-filenames get even shorter—now use base 36 (0-9 and a-z) for times and volume numbers. This change is not backwards compatible but probably I’m the only one affected. If not, let me know and I can probably write a little script.
Improved the way files are packed into the volumes so less space is wasted per volume.
Decreased default volume size to 5MB (from 50MB) in preparation for the new restore system planned for 0.4.0.
Added ftp backend. Now the remote repository can accessed by ftp and can be specified like “ftp://email@example.com/”.
Added –ssh-command and –scp-command options, which can be used to replace ssh and scp, or pass different arguments to them. Requested by Will Dyson.
Added –short-filenames option, for use when uploading to a file system that can’t have filenames longer than 30 characters (e.g. Mac OS 8).
Added –list-current-files mode, which lists the files currently at the given destination url.
Ported some statistics code from rdiff-backup. Now by default statistics are printed after a backup session. This can be disabled with the –no-print-statistics option. Thanks to Mathieu Doidy for requesting the above two features.
Added rdiff-backup-style –include/exclude-filelist-globbing options. Now filelists with “+ /-” can be treated exactly like repeated –include or –exclude options.
Fixed bug backing up normal files instead of directories. Thanks to Mathieu Doidy for bug report.
Security Fix: Fixed bug where an rdiffdir patch containing a path with a ‘..’ component could overwrite files in parent directory.
Fixed bug where files without a username (only uid) would be marked as changed every session. Thanks to Peter Ehrenberg for bug report.
Now files with negative mtimes are treated as if they have mtime 0. This prevents them from being marked as changed unnecessarily. Thanks to Peter Ehrenberg for report.
(Thanks to Peter Ehrenberg for his valuable suggestions on many of the below.)
** IMPORTANT ** New syntax intended to make duplicity more user-friendly. Old duplicity invocation will not work so change your scripts and read the man page.
By default no local archive dir will be read or created. $HOME/.duplicity has no special significance. However, one can be specified with –archive-dir for faster operation.
If PASSPHRASE environment variable not set, ask user for passphrase instead of exiting.
The source directory and hostname are now recorded, and rdiff-backup will complain if you try to back up different directories to the same location (but –allow-source-mismatch can override).
Changed some of the logging verbosity levels. Level 5 now lists files being backed up.
Fixed large file problems. Files larger than 2GB should be backed up correctly.
Significant additions to the manual page.
By default, duplicity now chooses whether to make a full or incremental backup depending on whether up-to-date signatures can be found on the remote side or local archive directory. Overrideable with (-f/–full and -i/–incremental).
Supports asymmetric encryption with GPG public keys using the –encrypt-key option.
Added option –sign-key. When backing up, files are signed with the given key. When restoring, remote files are required to be signed by given key.
Now duplicity checks the hash data found in the manifest file. The previous version generated SHA1 hashes but did not check them when restoring.
Fixed GnuPG broken pipe error. In the previous version certain partial restores would end with a spurious GnuPG error message.
First real version.