/brz/remove-bazaar

To get this branch, use:
bzr branch http://gegoxaren.bato24.eu/bzr/brz/remove-bazaar
5752.3.8 by John Arbash Meinel
Merge bzr.dev 5764 to resolve release-notes (aka NEWS) conflicts
1
# Copyright (C) 2005-2011 Canonical Ltd
1887.1.1 by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines,
2
#
1 by mbp at sourcefrog
import from baz patch-364
3
# This program is free software; you can redistribute it and/or modify
4
# it under the terms of the GNU General Public License as published by
5
# the Free Software Foundation; either version 2 of the License, or
6
# (at your option) any later version.
1887.1.1 by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines,
7
#
1 by mbp at sourcefrog
import from baz patch-364
8
# This program is distributed in the hope that it will be useful,
9
# but WITHOUT ANY WARRANTY; without even the implied warranty of
10
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11
# GNU General Public License for more details.
1887.1.1 by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines,
12
#
1 by mbp at sourcefrog
import from baz patch-364
13
# You should have received a copy of the GNU General Public License
14
# along with this program; if not, write to the Free Software
4183.7.1 by Sabin Iacob
update FSF mailing address
15
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
1 by mbp at sourcefrog
import from baz patch-364
16
17
"""Tree classes, representing directory at point in time.
18
"""
19
849 by Martin Pool
- Put files inside an exported tarball into a top-level directory rather than
20
import os
5745.3.1 by Jelmer Vernooij
Use lazy imports in bzrlib.tree.
21
22
from bzrlib.lazy_import import lazy_import
23
lazy_import(globals(), """
5745.3.4 by Jelmer Vernooij
Merge bzr.dev.
24
import collections
800 by Martin Pool
Merge John's import-speedup branch:
25
2079.1.1 by John Arbash Meinel
Create a deprecated bzrlib.tree.RevisionTree() in favor of bzrlib.revisiontree.RevisionTree()
26
from bzrlib import (
3398.1.24 by Ian Clatworthy
make iter_search_rules a tree method
27
    conflicts as _mod_conflicts,
3368.2.30 by Ian Clatworthy
add -Dfilters support
28
    debug,
2079.1.1 by John Arbash Meinel
Create a deprecated bzrlib.tree.RevisionTree() in favor of bzrlib.revisiontree.RevisionTree()
29
    delta,
5745.3.1 by Jelmer Vernooij
Use lazy imports in bzrlib.tree.
30
    errors,
3368.2.16 by Ian Clatworthy
add real implementation of Tree.get_filter_stack
31
    filters,
5745.3.4 by Jelmer Vernooij
Merge bzr.dev.
32
    inventory,
2012.1.8 by Aaron Bentley
Merge from bzr.dev
33
    osutils,
1551.15.46 by Aaron Bentley
Move plan merge to tree
34
    revision as _mod_revision,
3398.1.24 by Ian Clatworthy
make iter_search_rules a tree method
35
    rules,
5745.3.4 by Jelmer Vernooij
Merge bzr.dev.
36
    trace,
2079.1.1 by John Arbash Meinel
Create a deprecated bzrlib.tree.RevisionTree() in favor of bzrlib.revisiontree.RevisionTree()
37
    )
5745.3.1 by Jelmer Vernooij
Use lazy imports in bzrlib.tree.
38
""")
39
1852.11.1 by Robert Collins
Deprecate compare_trees and move its body to InterTree.changes_from.
40
from bzrlib.decorators import needs_read_lock
1852.8.2 by Robert Collins
Add InterTree class to represent InterTree operations.
41
from bzrlib.inter import InterObject
5837.2.1 by Jelmer Vernooij
Deprecate Tree.__iter__.
42
from bzrlib.symbol_versioning import (
43
    deprecated_in,
44
    deprecated_method,
45
    )
1 by mbp at sourcefrog
import from baz patch-364
46
1852.5.1 by Robert Collins
Deprecate EmptyTree in favour of using Repository.revision_tree.
47
558 by Martin Pool
- All top-level classes inherit from object
48
class Tree(object):
1 by mbp at sourcefrog
import from baz patch-364
49
    """Abstract file tree.
50
51
    There are several subclasses:
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
52
1 by mbp at sourcefrog
import from baz patch-364
53
    * `WorkingTree` exists as files on disk editable by the user.
54
55
    * `RevisionTree` is a tree as recorded at some point in the past.
56
57
    Trees can be compared, etc, regardless of whether they are working
58
    trees or versioned trees.
59
    """
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
60
1852.9.6 by Robert Collins
Merge the change from Tree.compare to Tree.changes_from.
61
    def changes_from(self, other, want_unchanged=False, specific_files=None,
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
62
        extra_trees=None, require_versioned=False, include_root=False,
63
        want_unversioned=False):
1852.8.8 by Robert Collins
change Tree.compare to Tree.changes_from - its better for the common case.
64
        """Return a TreeDelta of the changes from other to this tree.
1852.9.3 by Robert Collins
Convert the test_delta tests to intertree_implementation and workingtree_implementation tests as appropriate.
65
66
        :param other: A tree to compare with.
67
        :param specific_files: An optional list of file paths to restrict the
68
            comparison to. When mapping filenames to ids, all matches in all
69
            trees (including optional extra_trees) are used, and all children of
70
            matched directories are included.
1852.9.4 by Robert Collins
Add minimal test for Tree.compare(extra_trees=...).
71
        :param want_unchanged: An optional boolean requesting the inclusion of
72
            unchanged entries in the result.
73
        :param extra_trees: An optional list of additional trees to use when
74
            mapping the contents of specific_files (paths) to file_ids.
1852.9.5 by Robert Collins
Add tests for require_versioned to the InterTree.compare() test suite.
75
        :param require_versioned: An optional boolean (defaults to False). When
76
            supplied and True all the 'specific_files' must be versioned, or
77
            a PathsNotVersionedError will be thrown.
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
78
        :param want_unversioned: Scan for unversioned paths.
1852.9.3 by Robert Collins
Convert the test_delta tests to intertree_implementation and workingtree_implementation tests as appropriate.
79
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
80
        The comparison will be performed by an InterTree object looked up on
1852.8.4 by Robert Collins
Hook InterTree into Tree.
81
        self and other.
82
        """
1852.8.8 by Robert Collins
change Tree.compare to Tree.changes_from - its better for the common case.
83
        # Martin observes that Tree.changes_from returns a TreeDelta and this
84
        # may confuse people, because the class name of the returned object is
85
        # a synonym of the object referenced in the method name.
1852.9.6 by Robert Collins
Merge the change from Tree.compare to Tree.changes_from.
86
        return InterTree.get(other, self).compare(
1852.9.4 by Robert Collins
Add minimal test for Tree.compare(extra_trees=...).
87
            want_unchanged=want_unchanged,
88
            specific_files=specific_files,
1852.9.5 by Robert Collins
Add tests for require_versioned to the InterTree.compare() test suite.
89
            extra_trees=extra_trees,
90
            require_versioned=require_versioned,
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
91
            include_root=include_root,
92
            want_unversioned=want_unversioned,
1852.9.5 by Robert Collins
Add tests for require_versioned to the InterTree.compare() test suite.
93
            )
2012.1.1 by Aaron Bentley
Implement change iterator
94
3254.1.1 by Aaron Bentley
Make Tree.iter_changes a public method
95
    def iter_changes(self, from_tree, include_unchanged=False,
2255.2.149 by Robert Collins
Crufty but existing _iter_changes implementation for WorkingTreeFormat4.
96
                     specific_files=None, pb=None, extra_trees=None,
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
97
                     require_versioned=True, want_unversioned=False):
4988.10.6 by John Arbash Meinel
Fix bug #304182 by adding a trivial docstring to Tree.iter_changes
98
        """See InterTree.iter_changes"""
2012.1.5 by Aaron Bentley
Implement specific file id and dangling id handling
99
        intertree = InterTree.get(from_tree, self)
3254.1.1 by Aaron Bentley
Make Tree.iter_changes a public method
100
        return intertree.iter_changes(include_unchanged, specific_files, pb,
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
101
            extra_trees, require_versioned, want_unversioned=want_unversioned)
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
102
1773.2.1 by Robert Collins
Teach all trees about unknowns, conflicts and get_parent_ids.
103
    def conflicts(self):
104
        """Get a list of the conflicts in the tree.
105
106
        Each conflict is an instance of bzrlib.conflicts.Conflict.
107
        """
2748.2.1 by Lukáš Lalinsky
Return ConflictsList() instead of [] from Tree.conflicts.
108
        return _mod_conflicts.ConflictList()
1773.2.1 by Robert Collins
Teach all trees about unknowns, conflicts and get_parent_ids.
109
2255.7.91 by Robert Collins
Move unknown detection in long status into the delta creation, saving a tree-scan.
110
    def extras(self):
111
        """For trees that can have unversioned files, return all such paths."""
112
        return []
113
1773.2.1 by Robert Collins
Teach all trees about unknowns, conflicts and get_parent_ids.
114
    def get_parent_ids(self):
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
115
        """Get the parent ids for this tree.
1773.2.1 by Robert Collins
Teach all trees about unknowns, conflicts and get_parent_ids.
116
117
        :return: a list of parent ids. [] is returned to indicate
118
        a tree with no parents.
119
        :raises: BzrError if the parents are not known.
120
        """
121
        raise NotImplementedError(self.get_parent_ids)
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
122
1 by mbp at sourcefrog
import from baz patch-364
123
    def has_filename(self, filename):
124
        """True if the tree has given filename."""
2818.2.1 by Ian Clatworthy
minor tree & dirstate code cleanups
125
        raise NotImplementedError(self.has_filename)
1 by mbp at sourcefrog
import from baz patch-364
126
1185.12.39 by abentley
Propogated has_or_had_id to Tree
127
    def has_id(self, file_id):
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
128
        raise NotImplementedError(self.has_id)
1185.12.39 by abentley
Propogated has_or_had_id to Tree
129
3363.15.3 by Aaron Bentley
Change Tree.__contains__ to tolerate subclasssing
130
    def __contains__(self, file_id):
131
        return self.has_id(file_id)
1852.6.9 by Robert Collins
Add more test trees to the tree-implementations tests.
132
1185.12.39 by abentley
Propogated has_or_had_id to Tree
133
    def has_or_had_id(self, file_id):
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
134
        raise NotImplementedError(self.has_or_had_id)
1 by mbp at sourcefrog
import from baz patch-364
135
2255.7.97 by Robert Collins
Teach delta.report_changes about unversioned files, removing all inventory access during status --short.
136
    def is_ignored(self, filename):
137
        """Check whether the filename is ignored by this tree.
138
139
        :param filename: The relative filename within the tree.
140
        :return: True if the filename is ignored.
141
        """
142
        return False
143
3146.8.16 by Aaron Bentley
Updates from review
144
    def all_file_ids(self):
3146.8.2 by Aaron Bentley
Introduce iter_all_file_ids, to avoid hitting Inventory for this case
145
        """Iterate through all file ids, including ids for missing files."""
5837.2.1 by Jelmer Vernooij
Deprecate Tree.__iter__.
146
        raise NotImplementedError(self.all_file_ids)
3146.8.2 by Aaron Bentley
Introduce iter_all_file_ids, to avoid hitting Inventory for this case
147
1 by mbp at sourcefrog
import from baz patch-364
148
    def id2path(self, file_id):
2255.11.5 by Martin Pool
Tree.id2path should raise NoSuchId, not return None.
149
        """Return the path for a file id.
150
151
        :raises NoSuchId:
152
        """
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
153
        raise NotImplementedError(self.id2path)
1 by mbp at sourcefrog
import from baz patch-364
154
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
155
    def iter_entries_by_dir(self, specific_file_ids=None, yield_parents=False):
1852.6.9 by Robert Collins
Add more test trees to the tree-implementations tests.
156
        """Walk the tree in 'by_dir' order.
157
3363.2.23 by Aaron Bentley
Fix iter_entries_by_dir ordering
158
        This will yield each entry in the tree as a (path, entry) tuple.
159
        The order that they are yielded is:
160
161
        Directories are walked in a depth-first lexicographical order,
162
        however, whenever a directory is reached, all of its direct child
163
        nodes are yielded in  lexicographical order before yielding the
164
        grandchildren.
165
166
        For example, in the tree::
167
168
           a/
169
             b/
170
               c
171
             d/
172
               e
173
           f/
174
             g
175
3363.5.4 by Aaron Bentley
Fix iteration order of iter_entries_by_dir
176
        The yield order (ignoring root) would be::
5891.1.3 by Andrew Bennetts
Move docstring formatting fixes.
177
3363.2.23 by Aaron Bentley
Fix iter_entries_by_dir ordering
178
          a, f, a/b, a/d, a/b/c, a/d/e, f/g
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
179
180
        :param yield_parents: If True, yield the parents from the root leading
181
            down to specific_file_ids that have been requested. This has no
182
            impact if specific_file_ids is None.
1852.6.9 by Robert Collins
Add more test trees to the tree-implementations tests.
183
        """
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
184
        raise NotImplementedError(self.iter_entries_by_dir)
1852.6.9 by Robert Collins
Add more test trees to the tree-implementations tests.
185
5793.2.1 by Jelmer Vernooij
Add Tree.list_files stub.
186
    def list_files(self, include_root=False, from_dir=None, recursive=True):
187
        """List all files in this tree.
188
189
        :param include_root: Whether to include the entry for the tree root
190
        :param from_dir: Directory under which to list files
191
        :param recursive: Whether to list files recursively
192
        :return: iterator over tuples of (path, versioned, kind, file_id,
193
            inventory entry)
194
        """
195
        raise NotImplementedError(self.list_files)
196
2255.2.226 by Robert Collins
Get merge_nested finally working: change nested tree iterators to take file_ids, and ensure the right branch is connected to in the merge logic. May not be suitable for shared repositories yet.
197
    def iter_references(self):
4370.3.2 by Ian Clatworthy
apply jam's review feedback
198
        if self.supports_tree_reference():
199
            for path, entry in self.iter_entries_by_dir():
200
                if entry.kind == 'tree-reference':
201
                    yield path, entry.file_id
2100.3.27 by Aaron Bentley
Enable nested commits
202
1465 by Robert Collins
Bugfix the new pull --clobber to not generate spurious conflicts.
203
    def kind(self, file_id):
2255.2.159 by Martin Pool
reference-trees in dirstate pass all tests.
204
        raise NotImplementedError("Tree subclass %s must implement kind"
205
            % self.__class__.__name__)
1465 by Robert Collins
Bugfix the new pull --clobber to not generate spurious conflicts.
206
3146.8.4 by Aaron Bentley
Eliminate direct use of inventory from transform application
207
    def stored_kind(self, file_id):
208
        """File kind stored for this file_id.
209
3146.8.15 by Aaron Bentley
Cleanup and docs
210
        May not match kind on disk for working trees.  Always available
211
        for versioned files, even when the file itself is missing.
3146.8.4 by Aaron Bentley
Eliminate direct use of inventory from transform application
212
        """
213
        return self.kind(file_id)
214
2776.1.7 by Robert Collins
* New method on ``bzrlib.tree.Tree`` ``path_content_summary`` provides a
215
    def path_content_summary(self, path):
216
        """Get a summary of the information about path.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
217
4595.11.2 by Martin Pool
Clarify contract of path_content_summary
218
        All the attributes returned are for the canonical form, not the
219
        convenient form (if content filters are in use.)
220
2776.1.7 by Robert Collins
* New method on ``bzrlib.tree.Tree`` ``path_content_summary`` provides a
221
        :param path: A relative path within the tree.
222
        :return: A tuple containing kind, size, exec, sha1-or-link.
223
            Kind is always present (see tree.kind()).
4595.11.13 by Martin Pool
Remove get_kind_and_executable_by_path; go back to using plain path_content_summary
224
            size is present if kind is file and the size of the 
225
                canonical form can be cheaply determined, None otherwise.
2776.1.7 by Robert Collins
* New method on ``bzrlib.tree.Tree`` ``path_content_summary`` provides a
226
            exec is None unless kind is file and the platform supports the 'x'
227
                bit.
228
            sha1-or-link is the link target if kind is symlink, or the sha1 if
229
                it can be obtained without reading the file.
230
        """
231
        raise NotImplementedError(self.path_content_summary)
232
2255.2.226 by Robert Collins
Get merge_nested finally working: change nested tree iterators to take file_ids, and ensure the right branch is connected to in the merge logic. May not be suitable for shared repositories yet.
233
    def get_reference_revision(self, file_id, path=None):
2255.2.158 by Martin Pool
Most of the integration of dirstate and subtree
234
        raise NotImplementedError("Tree subclass %s must implement "
235
                                  "get_reference_revision"
236
            % self.__class__.__name__)
2100.3.20 by Aaron Bentley
Implement tree comparison for tree references
237
2012.1.7 by Aaron Bentley
Get tree._iter_changed down to ~ 1 stat per file
238
    def _comparison_data(self, entry, path):
2012.1.15 by Aaron Bentley
Minor tweaks
239
        """Return a tuple of kind, executable, stat_value for a file.
240
241
        entry may be None if there is no inventory entry for the file, but
242
        path must always be supplied.
243
244
        kind is None if there is no file present (even if an inventory id is
245
        present).  executable is False for non-file entries.
246
        """
2012.1.7 by Aaron Bentley
Get tree._iter_changed down to ~ 1 stat per file
247
        raise NotImplementedError(self._comparison_data)
248
2255.2.10 by Robert Collins
Now all tests matching dirstate pass - added generation of inventories for parent trees.
249
    def _file_size(self, entry, stat_value):
2012.1.7 by Aaron Bentley
Get tree._iter_changed down to ~ 1 stat per file
250
        raise NotImplementedError(self._file_size)
251
2743.3.3 by Ian Clatworthy
Skip path lookup for tree.get_file() when we already know the path
252
    def get_file(self, file_id, path=None):
2772.2.1 by Ian Clatworthy
(Ian Clatworthy) Quicker initial commit - skip SHAing twice & skip path lookup as we know it
253
        """Return a file object for the file file_id in the tree.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
254
2772.2.1 by Ian Clatworthy
(Ian Clatworthy) Quicker initial commit - skip SHAing twice & skip path lookup as we know it
255
        If both file_id and path are defined, it is implementation defined as
256
        to which one is used.
257
        """
1986.1.2 by Robert Collins
Various changes to allow non-workingtree specific tests to run entirely
258
        raise NotImplementedError(self.get_file)
2255.7.36 by John Arbash Meinel
All trees should implement get_file_mtime()
259
4354.4.5 by Aaron Bentley
Ensure Tree.get_file_with_stat is provided.
260
    def get_file_with_stat(self, file_id, path=None):
4354.4.7 by Aaron Bentley
Move MutableTree.get_file_with_stat to Tree.get_file_with_stat.
261
        """Get a file handle and stat object for file_id.
262
263
        The default implementation returns (self.get_file, None) for backwards
264
        compatibility.
265
266
        :param file_id: The file id to read.
267
        :param path: The path of the file, if it is known.
268
        :return: A tuple (file_handle, stat_value_or_None). If the tree has
269
            no stat facility, or need for a stat cache feedback during commit,
270
            it may return None for the second element of the tuple.
271
        """
272
        return (self.get_file(file_id, path), None)
4354.4.5 by Aaron Bentley
Ensure Tree.get_file_with_stat is provided.
273
3774.1.1 by Aaron Bentley
Test Tree.get_file_text() and supply default implementation.
274
    def get_file_text(self, file_id, path=None):
275
        """Return the byte content of a file.
276
277
        :param file_id: The file_id of the file.
278
        :param path: The path of the file.
279
        If both file_id and path are supplied, an implementation may use
280
        either one.
281
        """
282
        my_file = self.get_file(file_id, path)
283
        try:
284
            return my_file.read()
285
        finally:
286
            my_file.close()
287
3774.1.2 by Aaron Bentley
Test Tree.get_file_lines, provide a default implementation
288
    def get_file_lines(self, file_id, path=None):
289
        """Return the content of a file, as lines.
290
291
        :param file_id: The file_id of the file.
292
        :param path: The path of the file.
5891.1.3 by Andrew Bennetts
Move docstring formatting fixes.
293
3774.1.2 by Aaron Bentley
Test Tree.get_file_lines, provide a default implementation
294
        If both file_id and path are supplied, an implementation may use
295
        either one.
296
        """
297
        return osutils.split_lines(self.get_file_text(file_id, path))
298
5883.1.2 by Jelmer Vernooij
Support stat_value argument to get_file_sha1
299
    def get_file_sha1(self, file_id, path=None, stat_value=None):
5777.1.3 by Jelmer Vernooij
Add stub for Tree.get_file_sha1.
300
        """Return the SHA1 file for a file.
301
302
        :param file_id: The handle for this file.
303
        :param path: The path that this file can be found at.
304
            These must point to the same object.
5883.1.2 by Jelmer Vernooij
Support stat_value argument to get_file_sha1
305
        :param stat_value: Optional stat value for the object
5777.1.3 by Jelmer Vernooij
Add stub for Tree.get_file_sha1.
306
        """
307
        raise NotImplementedError(self.get_file_sha1)
308
2255.7.36 by John Arbash Meinel
All trees should implement get_file_mtime()
309
    def get_file_mtime(self, file_id, path=None):
310
        """Return the modification time for a file.
311
312
        :param file_id: The handle for this file.
313
        :param path: The path that this file can be found at.
314
            These must point to the same object.
315
        """
316
        raise NotImplementedError(self.get_file_mtime)
317
3363.3.4 by Aaron Bentley
Add get_file_size to Tree interface
318
    def get_file_size(self, file_id):
319
        """Return the size of a file in bytes.
320
321
        This applies only to regular files.  If invoked on directories or
322
        symlinks, it will return None.
323
        :param file_id: The file-id of the file
324
        """
325
        raise NotImplementedError(self.get_file_size)
326
974.1.26 by aaron.bentley at utoronto
merged mbp@sourcefrog.net-20050817233101-0939da1cf91f2472
327
    def get_file_by_path(self, path):
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
328
        raise NotImplementedError(self.get_file_by_path)
1 by mbp at sourcefrog
import from baz patch-364
329
5777.1.4 by Jelmer Vernooij
Stub for is_executable.
330
    def is_executable(self, file_id, path=None):
331
        """Check if a file is executable.
332
333
        :param file_id: The handle for this file.
334
        :param path: The path that this file can be found at.
335
            These must point to the same object.
336
        """
337
        raise NotImplementedError(self.is_executable)
338
2708.1.7 by Aaron Bentley
Rename extract_files_bytes to iter_files_bytes
339
    def iter_files_bytes(self, desired_files):
2708.1.6 by Aaron Bentley
Turn extract_files_bytes into an iterator
340
        """Iterate through file contents.
2708.1.1 by Aaron Bentley
Implement Tree.extract_files
341
2708.1.10 by Aaron Bentley
Update docstrings
342
        Files will not necessarily be returned in the order they occur in
343
        desired_files.  No specific order is guaranteed.
344
345
        Yields pairs of identifier, bytes_iterator.  identifier is an opaque
346
        value supplied by the caller as part of desired_files.  It should
347
        uniquely identify the file version in the caller's context.  (Examples:
348
        an index number or a TreeTransform trans_id.)
349
350
        bytes_iterator is an iterable of bytestrings for the file.  The
351
        kind of iterable and length of the bytestrings are unspecified, but for
352
        this implementation, it is a tuple containing a single bytestring with
353
        the complete text of the file.
354
2708.1.6 by Aaron Bentley
Turn extract_files_bytes into an iterator
355
        :param desired_files: a list of (file_id, identifier) pairs
2708.1.1 by Aaron Bentley
Implement Tree.extract_files
356
        """
2708.1.6 by Aaron Bentley
Turn extract_files_bytes into an iterator
357
        for file_id, identifier in desired_files:
2708.1.10 by Aaron Bentley
Update docstrings
358
            # We wrap the string in a tuple so that we can return an iterable
359
            # of bytestrings.  (Technically, a bytestring is also an iterable
360
            # of bytestrings, but iterating through each character is not
361
            # performant.)
2708.1.6 by Aaron Bentley
Turn extract_files_bytes into an iterator
362
            cur_file = (self.get_file_text(file_id),)
363
            yield identifier, cur_file
2708.1.1 by Aaron Bentley
Implement Tree.extract_files
364
5858.1.1 by Jelmer Vernooij
Support optional path argument to Tree.get_symlink_target.
365
    def get_symlink_target(self, file_id, path=None):
2255.2.134 by John Arbash Meinel
Add a tree-test for get_symlink_target
366
        """Get the target for a given file_id.
367
368
        It is assumed that the caller already knows that file_id is referencing
369
        a symlink.
370
        :param file_id: Handle for the symlink entry.
5858.1.1 by Jelmer Vernooij
Support optional path argument to Tree.get_symlink_target.
371
        :param path: The path of the file.
372
        If both file_id and path are supplied, an implementation may use
373
        either one.
2255.2.134 by John Arbash Meinel
Add a tree-test for get_symlink_target
374
        :return: The path the symlink points to.
375
        """
376
        raise NotImplementedError(self.get_symlink_target)
377
2946.3.2 by John Arbash Meinel
Add tree implementation tests for Tree.get_root_id()
378
    def get_root_id(self):
379
        """Return the file_id for the root of this tree."""
380
        raise NotImplementedError(self.get_root_id)
381
3224.1.2 by John Arbash Meinel
Updated the base Tree.annotate_iter() since all implemenations take an optional kwarg.
382
    def annotate_iter(self, file_id,
383
                      default_revision=_mod_revision.CURRENT_REVISION):
2818.2.1 by Ian Clatworthy
minor tree & dirstate code cleanups
384
        """Return an iterator of revision_id, line tuples.
1551.9.18 by Aaron Bentley
Updates from review comments
385
1551.9.16 by Aaron Bentley
Implement Tree.annotate_iter for RevisionTree and WorkingTree
386
        For working trees (and mutable trees in general), the special
387
        revision_id 'current:' will be used for lines that are new in this
388
        tree, e.g. uncommitted changes.
389
        :param file_id: The file to produce an annotated version from
3224.1.2 by John Arbash Meinel
Updated the base Tree.annotate_iter() since all implemenations take an optional kwarg.
390
        :param default_revision: For lines that don't match a basis, mark them
391
            with this revision id. Not all implementations will make use of
392
            this value.
1551.9.16 by Aaron Bentley
Implement Tree.annotate_iter for RevisionTree and WorkingTree
393
        """
394
        raise NotImplementedError(self.annotate_iter)
395
3144.3.1 by Aaron Bentley
Implement LCA merge, with problematic conflict markers
396
    def _get_plan_merge_data(self, file_id, other, base):
4496.3.15 by Andrew Bennetts
Remove unused imports.
397
        from bzrlib import versionedfile
3062.1.11 by Aaron Bentley
Update references
398
        vf = versionedfile._PlanMergeVersionedFile(file_id)
3062.1.3 by Aaron Bentley
Correctly determine file revisions
399
        last_revision_a = self._get_file_revision(file_id, vf, 'this:')
400
        last_revision_b = other._get_file_revision(file_id, vf, 'other:')
3062.2.4 by Aaron Bentley
Start supporting merge-with-base
401
        if base is None:
402
            last_revision_base = None
403
        else:
3062.2.6 by Aaron Bentley
Get cherrypick-on-weave working
404
            last_revision_base = base._get_file_revision(file_id, vf, 'base:')
3144.3.1 by Aaron Bentley
Implement LCA merge, with problematic conflict markers
405
        return vf, last_revision_a, last_revision_b, last_revision_base
406
407
    def plan_file_merge(self, file_id, other, base=None):
408
        """Generate a merge plan based on annotations.
409
410
        If the file contains uncommitted changes in this tree, they will be
411
        attributed to the 'current:' pseudo-revision.  If the file contains
412
        uncommitted changes in the other tree, they will be assigned to the
413
        'other:' pseudo-revision.
414
        """
415
        data = self._get_plan_merge_data(file_id, other, base)
416
        vf, last_revision_a, last_revision_b, last_revision_base = data
3062.2.4 by Aaron Bentley
Start supporting merge-with-base
417
        return vf.plan_merge(last_revision_a, last_revision_b,
418
                             last_revision_base)
3062.1.3 by Aaron Bentley
Correctly determine file revisions
419
3144.3.1 by Aaron Bentley
Implement LCA merge, with problematic conflict markers
420
    def plan_file_lca_merge(self, file_id, other, base=None):
421
        """Generate a merge plan based lca-newness.
422
423
        If the file contains uncommitted changes in this tree, they will be
424
        attributed to the 'current:' pseudo-revision.  If the file contains
425
        uncommitted changes in the other tree, they will be assigned to the
426
        'other:' pseudo-revision.
427
        """
428
        data = self._get_plan_merge_data(file_id, other, base)
429
        vf, last_revision_a, last_revision_b, last_revision_base = data
430
        return vf.plan_lca_merge(last_revision_a, last_revision_b,
431
                                 last_revision_base)
432
3363.9.1 by Aaron Bentley
Implement plan_merge, refactoring various bits
433
    def _iter_parent_trees(self):
434
        """Iterate through parent trees, defaulting to Tree.revision_tree."""
435
        for revision_id in self.get_parent_ids():
436
            try:
437
                yield self.revision_tree(revision_id)
438
            except errors.NoSuchRevisionInTree:
439
                yield self.repository.revision_tree(revision_id)
440
3062.1.3 by Aaron Bentley
Correctly determine file revisions
441
    def _get_file_revision(self, file_id, vf, tree_revision):
3350.6.4 by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores.
442
        """Ensure that file_id, tree_revision is in vf to plan the merge."""
3062.1.3 by Aaron Bentley
Correctly determine file revisions
443
3350.6.4 by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores.
444
        if getattr(self, '_repository', None) is None:
3062.1.3 by Aaron Bentley
Correctly determine file revisions
445
            last_revision = tree_revision
5793.2.3 by Jelmer Vernooij
Add a RevisionTree.get_file_revision() method.
446
            parent_keys = [(file_id, t.get_file_revision(file_id)) for t in
3363.9.1 by Aaron Bentley
Implement plan_merge, refactoring various bits
447
                self._iter_parent_trees()]
3350.6.4 by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores.
448
            vf.add_lines((file_id, last_revision), parent_keys,
4708.2.1 by Martin
Ensure all files opened by bazaar proper are explicitly closed
449
                         self.get_file_lines(file_id))
3062.1.6 by Aaron Bentley
PlanMergeVersionedfile now has multiple Versionedfile fallbacks
450
            repo = self.branch.repository
3350.6.4 by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores.
451
            base_vf = repo.texts
3062.1.3 by Aaron Bentley
Correctly determine file revisions
452
        else:
5793.2.3 by Jelmer Vernooij
Add a RevisionTree.get_file_revision() method.
453
            last_revision = self.get_file_revision(file_id)
3350.6.4 by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores.
454
            base_vf = self._repository.texts
455
        if base_vf not in vf.fallback_versionedfiles:
456
            vf.fallback_versionedfiles.append(base_vf)
3062.1.3 by Aaron Bentley
Correctly determine file revisions
457
        return last_revision
1551.15.46 by Aaron Bentley
Move plan merge to tree
458
1 by mbp at sourcefrog
import from baz patch-364
459
    def _check_retrieved(self, ie, f):
1364 by Martin Pool
- remove extra verification of files retrieved from tree
460
        if not __debug__:
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
461
            return
5745.3.1 by Jelmer Vernooij
Use lazy imports in bzrlib.tree.
462
        fp = osutils.fingerprint_file(f)
130 by mbp at sourcefrog
- fixup checks on retrieved files to cope with compression,
463
        f.seek(0)
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
464
1963.2.6 by Robey Pointer
pychecker is on crack; go back to using 'is None'.
465
        if ie.text_size is not None:
131 by mbp at sourcefrog
check size and sha1 of files retrieved from the tree
466
            if ie.text_size != fp['size']:
5753.2.2 by Jelmer Vernooij
Remove some unnecessary imports, clean up lazy imports.
467
                raise errors.BzrError(
468
                        "mismatched size for file %r in %r" %
5745.3.1 by Jelmer Vernooij
Use lazy imports in bzrlib.tree.
469
                        (ie.file_id, self._store),
1 by mbp at sourcefrog
import from baz patch-364
470
                        ["inventory expects %d bytes" % ie.text_size,
130 by mbp at sourcefrog
- fixup checks on retrieved files to cope with compression,
471
                         "file is actually %d bytes" % fp['size'],
1 by mbp at sourcefrog
import from baz patch-364
472
                         "store is probably damaged/corrupt"])
473
130 by mbp at sourcefrog
- fixup checks on retrieved files to cope with compression,
474
        if ie.text_sha1 != fp['sha1']:
5745.3.1 by Jelmer Vernooij
Use lazy imports in bzrlib.tree.
475
            raise errors.BzrError("wrong SHA-1 for file %r in %r" %
476
                    (ie.file_id, self._store),
1 by mbp at sourcefrog
import from baz patch-364
477
                    ["inventory expects %s" % ie.text_sha1,
130 by mbp at sourcefrog
- fixup checks on retrieved files to cope with compression,
478
                     "file is actually %s" % fp['sha1'],
1 by mbp at sourcefrog
import from baz patch-364
479
                     "store is probably damaged/corrupt"])
480
1986.1.2 by Robert Collins
Various changes to allow non-workingtree specific tests to run entirely
481
    def path2id(self, path):
482
        """Return the id for path in this tree."""
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
483
        raise NotImplementedError(self.path2id)
1 by mbp at sourcefrog
import from baz patch-364
484
2255.2.101 by Robert Collins
Finish making Tree.ids2paths support the file_ids_across_trees api.
485
    def paths2ids(self, paths, trees=[], require_versioned=True):
2255.2.100 by Robert Collins
Create a paths2ids api to replace find_ids_across_trees, with tests.
486
        """Return all the ids that can be reached by walking from paths.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
487
2818.2.1 by Ian Clatworthy
minor tree & dirstate code cleanups
488
        Each path is looked up in this tree and any extras provided in
2255.2.100 by Robert Collins
Create a paths2ids api to replace find_ids_across_trees, with tests.
489
        trees, and this is repeated recursively: the children in an extra tree
490
        of a directory that has been renamed under a provided path in this tree
2818.2.1 by Ian Clatworthy
minor tree & dirstate code cleanups
491
        are all returned, even if none exist under a provided path in this
2255.2.100 by Robert Collins
Create a paths2ids api to replace find_ids_across_trees, with tests.
492
        tree, and vice versa.
2255.2.101 by Robert Collins
Finish making Tree.ids2paths support the file_ids_across_trees api.
493
494
        :param paths: An iterable of paths to start converting to ids from.
2255.2.104 by Robert Collins
Add WorkingTree4.paths2ids which is inventory-usage free if the trees being examined are in the dirstate.
495
            Alternatively, if paths is None, no ids should be calculated and None
496
            will be returned. This is offered to make calling the api unconditional
497
            for code that *might* take a list of files.
2255.2.101 by Robert Collins
Finish making Tree.ids2paths support the file_ids_across_trees api.
498
        :param trees: Additional trees to consider.
499
        :param require_versioned: If False, do not raise NotVersionedError if
500
            an element of paths is not versioned in this tree and all of trees.
2255.2.100 by Robert Collins
Create a paths2ids api to replace find_ids_across_trees, with tests.
501
        """
2255.2.105 by Robert Collins
Unfuck InterTree.compare which I broke with the paths2ids implementation.
502
        return find_ids_across_trees(paths, [self] + list(trees), require_versioned)
2255.2.100 by Robert Collins
Create a paths2ids api to replace find_ids_across_trees, with tests.
503
3363.12.2 by Aaron Bentley
Implement tree.iter_children to instead of adjusting InventoryEntry handling
504
    def iter_children(self, file_id):
505
        entry = self.iter_entries_by_dir([file_id]).next()[1]
506
        for child in getattr(entry, 'children', {}).itervalues():
507
            yield child.file_id
508
1543.1.1 by Denys Duchier
lock operations for trees - use them for diff
509
    def lock_read(self):
5200.3.6 by Robert Collins
Make all lock methods return Result objects, rather than lock_read returning self, as per John's review.
510
        """Lock this tree for multiple read only operations.
511
        
512
        :return: A bzrlib.lock.LogicalLockResult.
513
        """
1543.1.1 by Denys Duchier
lock operations for trees - use them for diff
514
        pass
515
1908.11.1 by Robert Collins
Add a new method ``Tree.revision_tree`` which allows access to cached
516
    def revision_tree(self, revision_id):
517
        """Obtain a revision tree for the revision revision_id.
518
519
        The intention of this method is to allow access to possibly cached
520
        tree data. Implementors of this method should raise NoSuchRevision if
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
521
        the tree is not locally available, even if they could obtain the
522
        tree via a repository or some other means. Callers are responsible
1908.11.1 by Robert Collins
Add a new method ``Tree.revision_tree`` which allows access to cached
523
        for finding the ultimate source for a revision tree.
524
525
        :param revision_id: The revision_id of the requested tree.
526
        :return: A Tree.
527
        :raises: NoSuchRevision if the tree cannot be obtained.
528
        """
529
        raise errors.NoSuchRevisionInTree(self, revision_id)
530
1773.2.1 by Robert Collins
Teach all trees about unknowns, conflicts and get_parent_ids.
531
    def unknowns(self):
532
        """What files are present in this tree and unknown.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
533
1773.2.1 by Robert Collins
Teach all trees about unknowns, conflicts and get_parent_ids.
534
        :return: an iterator over the unknown files.
535
        """
536
        return iter([])
537
1543.1.1 by Denys Duchier
lock operations for trees - use them for diff
538
    def unlock(self):
539
        pass
1658.1.9 by Martin Pool
Give an error for bzr diff on an nonexistent file (Malone #3619)
540
541
    def filter_unversioned_files(self, paths):
2255.7.62 by Robert Collins
Update the Tree.filter_unversioned_files docstring to reflect what the existing implementations actually do, and change the WorkingTree4 implementation to match a newly created test for it.
542
        """Filter out paths that are versioned.
1658.1.9 by Martin Pool
Give an error for bzr diff on an nonexistent file (Malone #3619)
543
544
        :return: set of paths.
545
        """
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
546
        raise NotImplementedError(self.filter_unversioned_files)
1852.7.1 by Robert Collins
Move RevisionTree out of tree.py.
547
1852.15.3 by Robert Collins
Add a first-cut Tree.walkdirs method.
548
    def walkdirs(self, prefix=""):
549
        """Walk the contents of this tree from path down.
550
551
        This yields all the data about the contents of a directory at a time.
552
        After each directory has been yielded, if the caller has mutated the
553
        list to exclude some directories, they are then not descended into.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
554
1852.15.3 by Robert Collins
Add a first-cut Tree.walkdirs method.
555
        The data yielded is of the form:
1852.15.7 by Robert Collins
Start testing behaviour of unknowns in WorkingTree.walkdirs.
556
        ((directory-relpath, directory-path-from-root, directory-fileid),
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
557
        [(relpath, basename, kind, lstat, path_from_tree_root, file_id,
1852.15.7 by Robert Collins
Start testing behaviour of unknowns in WorkingTree.walkdirs.
558
          versioned_kind), ...]),
559
         - directory-relpath is the containing dirs relpath from prefix
560
         - directory-path-from-root is the containing dirs path from /
561
         - directory-fileid is the id of the directory if it is versioned.
1852.15.3 by Robert Collins
Add a first-cut Tree.walkdirs method.
562
         - relpath is the relative path within the subtree being walked.
563
         - basename is the basename
564
         - kind is the kind of the file now. If unknonwn then the file is not
565
           present within the tree - but it may be recorded as versioned. See
566
           versioned_kind.
567
         - lstat is the stat data *if* the file was statted.
568
         - path_from_tree_root is the path from the root of the tree.
2818.2.1 by Ian Clatworthy
minor tree & dirstate code cleanups
569
         - file_id is the file_id if the entry is versioned.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
570
         - versioned_kind is the kind of the file as last recorded in the
1852.15.3 by Robert Collins
Add a first-cut Tree.walkdirs method.
571
           versioning system. If 'unknown' the file is not versioned.
572
        One of 'kind' and 'versioned_kind' must not be 'unknown'.
573
574
        :param prefix: Start walking from prefix within the tree rather than
575
        at the root. This allows one to walk a subtree but get paths that are
576
        relative to a tree rooted higher up.
577
        :return: an iterator over the directory data.
578
        """
579
        raise NotImplementedError(self.walkdirs)
580
3368.2.45 by Ian Clatworthy
add and use supports_content_filtering API
581
    def supports_content_filtering(self):
582
        return False
583
3368.2.5 by Ian Clatworthy
incorporate jameinel's review feedback
584
    def _content_filter_stack(self, path=None, file_id=None):
3368.2.45 by Ian Clatworthy
add and use supports_content_filtering API
585
        """The stack of content filters for a path if filtering is supported.
3368.2.47 by Ian Clatworthy
merge bzr.dev r4042
586
3368.2.4 by Ian Clatworthy
make content filter lookup a tree responsibility
587
        Readers will be applied in first-to-last order.
588
        Writers will be applied in last-to-first order.
3368.2.5 by Ian Clatworthy
incorporate jameinel's review feedback
589
        Either the path or the file-id needs to be provided.
590
591
        :param path: path relative to the root of the tree
592
            or None if unknown
593
        :param file_id: file_id or None if unknown
594
        :return: the list of filters - [] if there are none
3368.2.4 by Ian Clatworthy
make content filter lookup a tree responsibility
595
        """
3368.2.16 by Ian Clatworthy
add real implementation of Tree.get_filter_stack
596
        filter_pref_names = filters._get_registered_names()
597
        if len(filter_pref_names) == 0:
598
            return []
599
        if path is None:
600
            path = self.id2path(file_id)
3368.2.29 by Ian Clatworthy
access rules via get_special_file to fix recursion bug
601
        prefs = self.iter_search_rules([path], filter_pref_names).next()
3368.2.30 by Ian Clatworthy
add -Dfilters support
602
        stk = filters._get_filter_stack_for(prefs)
603
        if 'filters' in debug.debug_flags:
5745.3.4 by Jelmer Vernooij
Merge bzr.dev.
604
            trace.note("*** %s content-filter: %s => %r" % (path,prefs,stk))
3368.2.30 by Ian Clatworthy
add -Dfilters support
605
        return stk
3368.2.4 by Ian Clatworthy
make content filter lookup a tree responsibility
606
3368.2.45 by Ian Clatworthy
add and use supports_content_filtering API
607
    def _content_filter_stack_provider(self):
608
        """A function that returns a stack of ContentFilters.
609
610
        The function takes a path (relative to the top of the tree) and a
611
        file-id as parameters.
612
613
        :return: None if content filtering is not supported by this tree.
614
        """
615
        if self.supports_content_filtering():
616
            return lambda path, file_id: \
617
                    self._content_filter_stack(path, file_id)
618
        else:
619
            return None
620
3398.1.24 by Ian Clatworthy
make iter_search_rules a tree method
621
    def iter_search_rules(self, path_names, pref_names=None,
4324.4.1 by Marius Kruger
Make it possible to blackboxtest rules
622
        _default_searcher=None):
3398.1.24 by Ian Clatworthy
make iter_search_rules a tree method
623
        """Find the preferences for filenames in a tree.
624
625
        :param path_names: an iterable of paths to find attributes for.
626
          Paths are given relative to the root of the tree.
627
        :param pref_names: the list of preferences to lookup - None for all
628
        :param _default_searcher: private parameter to assist testing - don't use
629
        :return: an iterator of tuple sequences, one per path-name.
630
          See _RulesSearcher.get_items for details on the tuple sequence.
631
        """
4324.4.1 by Marius Kruger
Make it possible to blackboxtest rules
632
        if _default_searcher is None:
633
            _default_searcher = rules._per_user_searcher
3398.1.24 by Ian Clatworthy
make iter_search_rules a tree method
634
        searcher = self._get_rules_searcher(_default_searcher)
635
        if searcher is not None:
3398.1.34 by Ian Clatworthy
changed API design as requested by jam during review
636
            if pref_names is not None:
637
                for path in path_names:
638
                    yield searcher.get_selected_items(path, pref_names)
639
            else:
640
                for path in path_names:
641
                    yield searcher.get_items(path)
3398.1.24 by Ian Clatworthy
make iter_search_rules a tree method
642
643
    def _get_rules_searcher(self, default_searcher):
644
        """Get the RulesSearcher for this tree given the default one."""
645
        searcher = default_searcher
646
        return searcher
647
1852.7.1 by Robert Collins
Move RevisionTree out of tree.py.
648
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
649
class InventoryTree(Tree):
650
    """A tree that relies on an inventory for its metadata.
651
652
    Trees contain an `Inventory` object, and also know how to retrieve
653
    file texts mentioned in the inventory, either from a working
654
    directory or from a store.
655
656
    It is possible for trees to contain files that are not described
657
    in their inventory or vice versa; for this use `filenames()`.
5777.3.5 by Jelmer Vernooij
Add note about _inventory.
658
659
    Subclasses should set the _inventory attribute, which is considered
660
    private to external API users.
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
661
    """
662
663
    def get_canonical_inventory_paths(self, paths):
664
        """Like get_canonical_inventory_path() but works on multiple items.
665
666
        :param paths: A sequence of paths relative to the root of the tree.
667
        :return: A list of paths, with each item the corresponding input path
668
        adjusted to account for existing elements that match case
669
        insensitively.
670
        """
671
        return list(self._yield_canonical_inventory_paths(paths))
672
673
    def get_canonical_inventory_path(self, path):
674
        """Returns the first inventory item that case-insensitively matches path.
675
676
        If a path matches exactly, it is returned. If no path matches exactly
677
        but more than one path matches case-insensitively, it is implementation
678
        defined which is returned.
679
680
        If no path matches case-insensitively, the input path is returned, but
681
        with as many path entries that do exist changed to their canonical
682
        form.
683
684
        If you need to resolve many names from the same tree, you should
685
        use get_canonical_inventory_paths() to avoid O(N) behaviour.
686
687
        :param path: A paths relative to the root of the tree.
688
        :return: The input path adjusted to account for existing elements
689
        that match case insensitively.
690
        """
691
        return self._yield_canonical_inventory_paths([path]).next()
692
693
    def _yield_canonical_inventory_paths(self, paths):
694
        for path in paths:
695
            # First, if the path as specified exists exactly, just use it.
696
            if self.path2id(path) is not None:
697
                yield path
698
                continue
699
            # go walkin...
700
            cur_id = self.get_root_id()
701
            cur_path = ''
702
            bit_iter = iter(path.split("/"))
703
            for elt in bit_iter:
704
                lelt = elt.lower()
705
                new_path = None
706
                for child in self.iter_children(cur_id):
707
                    try:
708
                        # XXX: it seem like if the child is known to be in the
709
                        # tree, we shouldn't need to go from its id back to
710
                        # its path -- mbp 2010-02-11
711
                        #
712
                        # XXX: it seems like we could be more efficient
713
                        # by just directly looking up the original name and
714
                        # only then searching all children; also by not
715
                        # chopping paths so much. -- mbp 2010-02-11
716
                        child_base = os.path.basename(self.id2path(child))
717
                        if (child_base == elt):
718
                            # if we found an exact match, we can stop now; if
719
                            # we found an approximate match we need to keep
720
                            # searching because there might be an exact match
721
                            # later.  
722
                            cur_id = child
723
                            new_path = osutils.pathjoin(cur_path, child_base)
724
                            break
725
                        elif child_base.lower() == lelt:
726
                            cur_id = child
727
                            new_path = osutils.pathjoin(cur_path, child_base)
728
                    except errors.NoSuchId:
729
                        # before a change is committed we can see this error...
730
                        continue
731
                if new_path:
732
                    cur_path = new_path
733
                else:
734
                    # got to the end of this directory and no entries matched.
735
                    # Return what matched so far, plus the rest as specified.
736
                    cur_path = osutils.pathjoin(cur_path, elt, *list(bit_iter))
737
                    break
738
            yield cur_path
739
        # all done.
740
741
    def _get_inventory(self):
742
        return self._inventory
743
744
    inventory = property(_get_inventory,
745
                         doc="Inventory of this Tree")
746
747
    @needs_read_lock
748
    def path2id(self, path):
749
        """Return the id for path in this tree."""
750
        return self._inventory.path2id(path)
751
752
    def id2path(self, file_id):
753
        """Return the path for a file id.
754
755
        :raises NoSuchId:
756
        """
757
        return self.inventory.id2path(file_id)
758
759
    def has_id(self, file_id):
760
        return self.inventory.has_id(file_id)
761
762
    def has_or_had_id(self, file_id):
763
        return self.inventory.has_id(file_id)
764
5837.2.1 by Jelmer Vernooij
Deprecate Tree.__iter__.
765
    def all_file_ids(self):
766
        return set(self.inventory)
767
768
    @deprecated_method(deprecated_in((2, 4, 0)))
5777.3.1 by Jelmer Vernooij
Split InventoryTree out of Tree.
769
    def __iter__(self):
770
        return iter(self.inventory)
771
772
    def filter_unversioned_files(self, paths):
773
        """Filter out paths that are versioned.
774
775
        :return: set of paths.
776
        """
777
        # NB: we specifically *don't* call self.has_filename, because for
778
        # WorkingTrees that can indicate files that exist on disk but that
779
        # are not versioned.
780
        pred = self.inventory.has_filename
781
        return set((p for p in paths if not pred(p)))
782
783
    @needs_read_lock
784
    def iter_entries_by_dir(self, specific_file_ids=None, yield_parents=False):
785
        """Walk the tree in 'by_dir' order.
786
787
        This will yield each entry in the tree as a (path, entry) tuple.
788
        The order that they are yielded is:
789
790
        See Tree.iter_entries_by_dir for details.
791
792
        :param yield_parents: If True, yield the parents from the root leading
793
            down to specific_file_ids that have been requested. This has no
794
            impact if specific_file_ids is None.
795
        """
796
        return self.inventory.iter_entries_by_dir(
797
            specific_file_ids=specific_file_ids, yield_parents=yield_parents)
798
799
    def get_file_by_path(self, path):
800
        return self.get_file(self._inventory.path2id(path), path)
801
802
1 by mbp at sourcefrog
import from baz patch-364
803
######################################################################
804
# diff
805
806
# TODO: Merge these two functions into a single one that can operate
807
# on either a whole tree or a set of files.
808
809
# TODO: Return the diff in order by filename, not by category or in
810
# random order.  Can probably be done by lock-stepping through the
811
# filenames from both trees.
812
813
814
def file_status(filename, old_tree, new_tree):
815
    """Return single-letter status, old and new names for a file.
816
817
    The complexity here is in deciding how to represent renames;
818
    many complex cases are possible.
819
    """
820
    old_inv = old_tree.inventory
821
    new_inv = new_tree.inventory
822
    new_id = new_inv.path2id(filename)
823
    old_id = old_inv.path2id(filename)
824
825
    if not new_id and not old_id:
826
        # easy: doesn't exist in either; not versioned at all
827
        if new_tree.is_ignored(filename):
828
            return 'I', None, None
829
        else:
830
            return '?', None, None
831
    elif new_id:
832
        # There is now a file of this name, great.
833
        pass
834
    else:
835
        # There is no longer a file of this name, but we can describe
836
        # what happened to the file that used to have
837
        # this name.  There are two possibilities: either it was
838
        # deleted entirely, or renamed.
839
        if new_inv.has_id(old_id):
840
            return 'X', old_inv.id2path(old_id), new_inv.id2path(old_id)
841
        else:
842
            return 'D', old_inv.id2path(old_id), None
843
844
    # if the file_id is new in this revision, it is added
845
    if new_id and not old_inv.has_id(new_id):
846
        return 'A'
847
848
    # if there used to be a file of this name, but that ID has now
849
    # disappeared, it is deleted
850
    if old_id and not new_inv.has_id(old_id):
851
        return 'D'
852
853
    return 'wtf?'
854
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
855
1551.7.22 by Aaron Bentley
Changes from review
856
def find_ids_across_trees(filenames, trees, require_versioned=True):
857
    """Find the ids corresponding to specified filenames.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
858
1551.7.22 by Aaron Bentley
Changes from review
859
    All matches in all trees will be used, and all children of matched
860
    directories will be used.
861
1551.10.7 by Aaron Bentley
Use new-style output for status
862
    :param filenames: The filenames to find file_ids for (if None, returns
863
        None)
1551.7.22 by Aaron Bentley
Changes from review
864
    :param trees: The trees to find file_ids within
865
    :param require_versioned: if true, all specified filenames must occur in
5891.1.3 by Andrew Bennetts
Move docstring formatting fixes.
866
        at least one tree.
1551.7.22 by Aaron Bentley
Changes from review
867
    :return: a set of file ids for the specified filenames and their children.
868
    """
3363.12.9 by Aaron Bentley
Revert changed select-all test
869
    if not filenames:
1551.7.22 by Aaron Bentley
Changes from review
870
        return None
2255.2.82 by Robert Collins
various notes about find_ids_across_trees
871
    specified_path_ids = _find_ids_across_trees(filenames, trees,
872
        require_versioned)
873
    return _find_children_across_trees(specified_path_ids, trees)
874
875
876
def _find_ids_across_trees(filenames, trees, require_versioned):
1551.7.22 by Aaron Bentley
Changes from review
877
    """Find the ids corresponding to specified filenames.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
878
2255.2.82 by Robert Collins
various notes about find_ids_across_trees
879
    All matches in all trees will be used, but subdirectories are not scanned.
1551.7.22 by Aaron Bentley
Changes from review
880
1551.7.14 by Aaron Bentley
Use specified_file_ids instead of is_inside_any in compare_trees
881
    :param filenames: The filenames to find file_ids for
882
    :param trees: The trees to find file_ids within
1551.7.16 by Aaron Bentley
Fix docs
883
    :param require_versioned: if true, all specified filenames must occur in
2255.2.82 by Robert Collins
various notes about find_ids_across_trees
884
        at least one tree.
2818.2.1 by Ian Clatworthy
minor tree & dirstate code cleanups
885
    :return: a set of file ids for the specified filenames
1551.7.14 by Aaron Bentley
Use specified_file_ids instead of is_inside_any in compare_trees
886
    """
1551.7.17 by Aaron Bentley
Switch to PathsNotVersioned, accept extra_trees
887
    not_versioned = []
1551.7.18 by Aaron Bentley
Indentation and documentation fixes
888
    interesting_ids = set()
889
    for tree_path in filenames:
890
        not_found = True
891
        for tree in trees:
2255.2.82 by Robert Collins
various notes about find_ids_across_trees
892
            file_id = tree.path2id(tree_path)
1551.7.18 by Aaron Bentley
Indentation and documentation fixes
893
            if file_id is not None:
894
                interesting_ids.add(file_id)
895
                not_found = False
896
        if not_found:
897
            not_versioned.append(tree_path)
1551.7.22 by Aaron Bentley
Changes from review
898
    if len(not_versioned) > 0 and require_versioned:
899
        raise errors.PathsNotVersionedError(not_versioned)
900
    return interesting_ids
901
902
903
def _find_children_across_trees(specified_ids, trees):
2818.2.1 by Ian Clatworthy
minor tree & dirstate code cleanups
904
    """Return a set including specified ids and their children.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
905
1551.7.22 by Aaron Bentley
Changes from review
906
    All matches in all trees will be used.
907
908
    :param trees: The trees to find file_ids within
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
909
    :return: a set containing all specified ids and their children
1551.7.22 by Aaron Bentley
Changes from review
910
    """
911
    interesting_ids = set(specified_ids)
1551.7.18 by Aaron Bentley
Indentation and documentation fixes
912
    pending = interesting_ids
913
    # now handle children of interesting ids
914
    # we loop so that we handle all children of each id in both trees
915
    while len(pending) > 0:
916
        new_pending = set()
917
        for file_id in pending:
1551.7.14 by Aaron Bentley
Use specified_file_ids instead of is_inside_any in compare_trees
918
            for tree in trees:
4544.2.1 by Robert Collins
Add interface enforcement for the behaviour of iter_changes with missing subtrees with explicit paths - the whole subtree is returned.
919
                if not tree.has_or_had_id(file_id):
1551.7.18 by Aaron Bentley
Indentation and documentation fixes
920
                    continue
3363.12.2 by Aaron Bentley
Implement tree.iter_children to instead of adjusting InventoryEntry handling
921
                for child_id in tree.iter_children(file_id):
922
                    if child_id not in interesting_ids:
923
                        new_pending.add(child_id)
1551.7.18 by Aaron Bentley
Indentation and documentation fixes
924
        interesting_ids.update(new_pending)
925
        pending = new_pending
1551.7.14 by Aaron Bentley
Use specified_file_ids instead of is_inside_any in compare_trees
926
    return interesting_ids
1852.8.2 by Robert Collins
Add InterTree class to represent InterTree operations.
927
928
929
class InterTree(InterObject):
930
    """This class represents operations taking place between two Trees.
931
932
    Its instances have methods like 'compare' and contain references to the
933
    source and target trees these operations are to be carried out on.
934
2818.2.1 by Ian Clatworthy
minor tree & dirstate code cleanups
935
    Clients of bzrlib should not need to use InterTree directly, rather they
1852.8.2 by Robert Collins
Add InterTree class to represent InterTree operations.
936
    should use the convenience methods on Tree such as 'Tree.compare()' which
937
    will pass through to InterTree as appropriate.
938
    """
939
4585.1.18 by Jelmer Vernooij
Add note saying that InterTree will not be tested if the from and to formats are not set.
940
    # Formats that will be used to test this InterTree. If both are
941
    # None, this InterTree will not be tested (e.g. because a complex
942
    # setup is required)
943
    _matching_from_tree_format = None
944
    _matching_to_tree_format = None
945
1910.2.15 by Aaron Bentley
Back out inter.get changes, make optimizers an ordered list
946
    _optimisers = []
1852.8.2 by Robert Collins
Add InterTree class to represent InterTree operations.
947
5837.1.2 by Jelmer Vernooij
Register defaults as first optimiser for existing Inter objects.
948
    @classmethod
949
    def is_compatible(kls, source, target):
950
        # The default implementation is naive and uses the public API, so
951
        # it works for all trees.
952
        return True
953
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
954
    def _changes_from_entries(self, source_entry, target_entry,
955
        source_path=None, target_path=None):
956
        """Generate a iter_changes tuple between source_entry and target_entry.
957
958
        :param source_entry: An inventory entry from self.source, or None.
959
        :param target_entry: An inventory entry from self.target, or None.
960
        :param source_path: The path of source_entry, if known. If not known
961
            it will be looked up.
962
        :param target_path: The path of target_entry, if known. If not known
963
            it will be looked up.
964
        :return: A tuple, item 0 of which is an iter_changes result tuple, and
965
            item 1 is True if there are any changes in the result tuple.
966
        """
967
        if source_entry is None:
968
            if target_entry is None:
969
                return None
970
            file_id = target_entry.file_id
971
        else:
972
            file_id = source_entry.file_id
973
        if source_entry is not None:
974
            source_versioned = True
975
            source_name = source_entry.name
976
            source_parent = source_entry.parent_id
977
            if source_path is None:
978
                source_path = self.source.id2path(file_id)
979
            source_kind, source_executable, source_stat = \
980
                self.source._comparison_data(source_entry, source_path)
981
        else:
982
            source_versioned = False
983
            source_name = None
984
            source_parent = None
985
            source_kind = None
986
            source_executable = None
987
        if target_entry is not None:
988
            target_versioned = True
989
            target_name = target_entry.name
990
            target_parent = target_entry.parent_id
991
            if target_path is None:
992
                target_path = self.target.id2path(file_id)
993
            target_kind, target_executable, target_stat = \
994
                self.target._comparison_data(target_entry, target_path)
995
        else:
996
            target_versioned = False
997
            target_name = None
998
            target_parent = None
999
            target_kind = None
1000
            target_executable = None
1001
        versioned = (source_versioned, target_versioned)
1002
        kind = (source_kind, target_kind)
1003
        changed_content = False
1004
        if source_kind != target_kind:
1005
            changed_content = True
1006
        elif source_kind == 'file':
1007
            if (self.source.get_file_sha1(file_id, source_path, source_stat) !=
1008
                self.target.get_file_sha1(file_id, target_path, target_stat)):
1009
                changed_content = True
1010
        elif source_kind == 'symlink':
1011
            if (self.source.get_symlink_target(file_id) !=
1012
                self.target.get_symlink_target(file_id)):
1013
                changed_content = True
1014
            # XXX: Yes, the indentation below is wrong. But fixing it broke
1015
            # test_merge.TestMergerEntriesLCAOnDisk.
1016
            # test_nested_tree_subtree_renamed_and_modified. We'll wait for
1017
            # the fix from bzr.dev -- vila 2009026
1018
            elif source_kind == 'tree-reference':
1019
                if (self.source.get_reference_revision(file_id, source_path)
1020
                    != self.target.get_reference_revision(file_id, target_path)):
1021
                    changed_content = True
1022
        parent = (source_parent, target_parent)
1023
        name = (source_name, target_name)
1024
        executable = (source_executable, target_executable)
1025
        if (changed_content is not False or versioned[0] != versioned[1]
1026
            or parent[0] != parent[1] or name[0] != name[1] or
1027
            executable[0] != executable[1]):
1028
            changes = True
1029
        else:
1030
            changes = False
1031
        return (file_id, (source_path, target_path), changed_content,
1032
                versioned, parent, name, kind, executable), changes
1033
1852.11.1 by Robert Collins
Deprecate compare_trees and move its body to InterTree.changes_from.
1034
    @needs_read_lock
1852.9.4 by Robert Collins
Add minimal test for Tree.compare(extra_trees=...).
1035
    def compare(self, want_unchanged=False, specific_files=None,
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
1036
        extra_trees=None, require_versioned=False, include_root=False,
1037
        want_unversioned=False):
1852.9.3 by Robert Collins
Convert the test_delta tests to intertree_implementation and workingtree_implementation tests as appropriate.
1038
        """Return the changes from source to target.
1852.8.3 by Robert Collins
Implement an InterTreeTestProvider and a trivial test_compare test case.
1039
1040
        :return: A TreeDelta.
1852.9.4 by Robert Collins
Add minimal test for Tree.compare(extra_trees=...).
1041
        :param specific_files: An optional list of file paths to restrict the
1042
            comparison to. When mapping filenames to ids, all matches in all
1043
            trees (including optional extra_trees) are used, and all children of
1044
            matched directories are included.
1045
        :param want_unchanged: An optional boolean requesting the inclusion of
1046
            unchanged entries in the result.
1047
        :param extra_trees: An optional list of additional trees to use when
1048
            mapping the contents of specific_files (paths) to file_ids.
1852.9.5 by Robert Collins
Add tests for require_versioned to the InterTree.compare() test suite.
1049
        :param require_versioned: An optional boolean (defaults to False). When
1050
            supplied and True all the 'specific_files' must be versioned, or
1051
            a PathsNotVersionedError will be thrown.
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
1052
        :param want_unversioned: Scan for unversioned paths.
1852.8.3 by Robert Collins
Implement an InterTreeTestProvider and a trivial test_compare test case.
1053
        """
2255.2.105 by Robert Collins
Unfuck InterTree.compare which I broke with the paths2ids implementation.
1054
        trees = (self.source,)
1852.11.1 by Robert Collins
Deprecate compare_trees and move its body to InterTree.changes_from.
1055
        if extra_trees is not None:
1056
            trees = trees + tuple(extra_trees)
2255.2.102 by Robert Collins
Use Tree.path2ids in status operations.
1057
        # target is usually the newer tree:
2255.2.105 by Robert Collins
Unfuck InterTree.compare which I broke with the paths2ids implementation.
1058
        specific_file_ids = self.target.paths2ids(specific_files, trees,
2255.2.102 by Robert Collins
Use Tree.path2ids in status operations.
1059
            require_versioned=require_versioned)
1852.11.1 by Robert Collins
Deprecate compare_trees and move its body to InterTree.changes_from.
1060
        if specific_files and not specific_file_ids:
1061
            # All files are unversioned, so just return an empty delta
1062
            # _compare_trees would think we want a complete delta
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
1063
            result = delta.TreeDelta()
5745.3.4 by Jelmer Vernooij
Merge bzr.dev.
1064
            fake_entry = inventory.InventoryFile('unused', 'unused', 'unused')
2255.7.91 by Robert Collins
Move unknown detection in long status into the delta creation, saving a tree-scan.
1065
            result.unversioned = [(path, None,
1066
                self.target._comparison_data(fake_entry, path)[0]) for path in
1067
                specific_files]
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
1068
            return result
1852.11.1 by Robert Collins
Deprecate compare_trees and move its body to InterTree.changes_from.
1069
        return delta._compare_trees(self.source, self.target, want_unchanged,
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
1070
            specific_files, include_root, extra_trees=extra_trees,
2655.2.1 by Marius Kruger
InterTree.compare and delta._compare_trees did not pass its
1071
            require_versioned=require_versioned,
2255.7.90 by Robert Collins
Add unversioned path reporting to TreeDelta.
1072
            want_unversioned=want_unversioned)
2012.1.1 by Aaron Bentley
Implement change iterator
1073
3254.1.1 by Aaron Bentley
Make Tree.iter_changes a public method
1074
    def iter_changes(self, include_unchanged=False,
2255.2.149 by Robert Collins
Crufty but existing _iter_changes implementation for WorkingTreeFormat4.
1075
                      specific_files=None, pb=None, extra_trees=[],
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1076
                      require_versioned=True, want_unversioned=False):
2012.1.1 by Aaron Bentley
Implement change iterator
1077
        """Generate an iterator of changes between trees.
1078
1079
        A tuple is returned:
2255.7.96 by Robert Collins
Change _iter_changes interface to yield both old and new paths.
1080
        (file_id, (path_in_source, path_in_target),
1081
         changed_content, versioned, parent, name, kind,
2012.1.1 by Aaron Bentley
Implement change iterator
1082
         executable)
1083
2255.7.96 by Robert Collins
Change _iter_changes interface to yield both old and new paths.
1084
        Changed_content is True if the file's content has changed.  This
1085
        includes changes to its kind, and to a symlink's target.
2012.1.1 by Aaron Bentley
Implement change iterator
1086
2012.1.15 by Aaron Bentley
Minor tweaks
1087
        versioned, parent, name, kind, executable are tuples of (from, to).
1088
        If a file is missing in a tree, its kind is None.
2012.1.1 by Aaron Bentley
Implement change iterator
1089
2255.2.122 by Robert Collins
Alter intertree implementation tests to let dirstate inter-trees be correctly parameterised.
1090
        Iteration is done in parent-to-child order, relative to the target
1091
        tree.
2255.2.149 by Robert Collins
Crufty but existing _iter_changes implementation for WorkingTreeFormat4.
1092
1093
        There is no guarantee that all paths are in sorted order: the
1094
        requirement to expand the search due to renames may result in children
1095
        that should be found early being found late in the search, after
1096
        lexically later results have been returned.
1097
        :param require_versioned: Raise errors.PathsNotVersionedError if a
1098
            path in the specific_files list is not versioned in one of
1099
            source, target or extra_trees.
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1100
        :param specific_files: An optional list of file paths to restrict the
1101
            comparison to. When mapping filenames to ids, all matches in all
1102
            trees (including optional extra_trees) are used, and all children
1103
            of matched directories are included. The parents in the target tree
1104
            of the specific files up to and including the root of the tree are
1105
            always evaluated for changes too.
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1106
        :param want_unversioned: Should unversioned files be returned in the
1107
            output. An unversioned file is defined as one with (False, False)
1108
            for the versioned pair.
2012.1.1 by Aaron Bentley
Implement change iterator
1109
        """
2255.2.149 by Robert Collins
Crufty but existing _iter_changes implementation for WorkingTreeFormat4.
1110
        lookup_trees = [self.source]
1111
        if extra_trees:
1112
             lookup_trees.extend(extra_trees)
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1113
        # The ids of items we need to examine to insure delta consistency.
1114
        precise_file_ids = set()
1115
        changed_file_ids = []
2748.3.1 by Aaron Bentley
Start supporting [] for empty list
1116
        if specific_files == []:
1117
            specific_file_ids = []
1118
        else:
1119
            specific_file_ids = self.target.paths2ids(specific_files,
1120
                lookup_trees, require_versioned=require_versioned)
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1121
        if specific_files is not None:
1122
            # reparented or added entries must have their parents included
1123
            # so that valid deltas can be created. The seen_parents set
1124
            # tracks the parents that we need to have.
1125
            # The seen_dirs set tracks directory entries we've yielded.
1126
            # After outputting version object in to_entries we set difference
1127
            # the two seen sets and start checking parents.
1128
            seen_parents = set()
1129
            seen_dirs = set()
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1130
        if want_unversioned:
2748.3.1 by Aaron Bentley
Start supporting [] for empty list
1131
            all_unversioned = sorted([(p.split('/'), p) for p in
1132
                                     self.target.extras()
1133
                if specific_files is None or
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1134
                    osutils.is_inside_any(specific_files, p)])
5745.3.4 by Jelmer Vernooij
Merge bzr.dev.
1135
            all_unversioned = collections.deque(all_unversioned)
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1136
        else:
5745.3.4 by Jelmer Vernooij
Merge bzr.dev.
1137
            all_unversioned = collections.deque()
2012.1.1 by Aaron Bentley
Implement change iterator
1138
        to_paths = {}
3363.14.2 by Aaron Bentley
Get iter_changes running to completion
1139
        from_entries_by_dir = list(self.source.iter_entries_by_dir(
1551.9.29 by Aaron Bentley
Optimize Tree._iter_changes with specific file_ids
1140
            specific_file_ids=specific_file_ids))
2012.1.16 by Aaron Bentley
Support progress bars in iter_changes
1141
        from_data = dict((e.file_id, (p, e)) for p, e in from_entries_by_dir)
3363.14.2 by Aaron Bentley
Get iter_changes running to completion
1142
        to_entries_by_dir = list(self.target.iter_entries_by_dir(
1551.9.31 by Aaron Bentley
Handle to_paths correctly when parent of file is not in specified_ids
1143
            specific_file_ids=specific_file_ids))
1551.9.29 by Aaron Bentley
Optimize Tree._iter_changes with specific file_ids
1144
        num_entries = len(from_entries_by_dir) + len(to_entries_by_dir)
2012.1.16 by Aaron Bentley
Support progress bars in iter_changes
1145
        entry_count = 0
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
1146
        # the unversioned path lookup only occurs on real trees - where there
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1147
        # can be extras. So the fake_entry is solely used to look up
1148
        # executable it values when execute is not supported.
5745.3.4 by Jelmer Vernooij
Merge bzr.dev.
1149
        fake_entry = inventory.InventoryFile('unused', 'unused', 'unused')
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1150
        for target_path, target_entry in to_entries_by_dir:
1151
            while (all_unversioned and
1152
                all_unversioned[0][0] < target_path.split('/')):
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1153
                unversioned_path = all_unversioned.popleft()
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1154
                target_kind, target_executable, target_stat = \
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1155
                    self.target._comparison_data(fake_entry, unversioned_path[1])
2255.7.96 by Robert Collins
Change _iter_changes interface to yield both old and new paths.
1156
                yield (None, (None, unversioned_path[1]), True, (False, False),
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1157
                    (None, None),
1158
                    (None, unversioned_path[0][-1]),
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1159
                    (None, target_kind),
1160
                    (None, target_executable))
1161
            source_path, source_entry = from_data.get(target_entry.file_id,
1162
                (None, None))
1163
            result, changes = self._changes_from_entries(source_entry,
1164
                target_entry, source_path=source_path, target_path=target_path)
1165
            to_paths[result[0]] = result[1][1]
2012.1.16 by Aaron Bentley
Support progress bars in iter_changes
1166
            entry_count += 1
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1167
            if result[3][0]:
1551.9.29 by Aaron Bentley
Optimize Tree._iter_changes with specific file_ids
1168
                entry_count += 1
2012.1.16 by Aaron Bentley
Support progress bars in iter_changes
1169
            if pb is not None:
1170
                pb.update('comparing files', entry_count, num_entries)
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1171
            if changes or include_unchanged:
1172
                if specific_file_ids is not None:
1173
                    new_parent_id = result[4][1]
1174
                    precise_file_ids.add(new_parent_id)
1175
                    changed_file_ids.append(result[0])
1176
                yield result
1177
            # Ensure correct behaviour for reparented/added specific files.
1178
            if specific_files is not None:
1179
                # Record output dirs
1180
                if result[6][1] == 'directory':
1181
                    seen_dirs.add(result[0])
1182
                # Record parents of reparented/added entries.
1183
                versioned = result[3]
1184
                parents = result[4]
1185
                if not versioned[0] or parents[0] != parents[1]:
1186
                    seen_parents.add(parents[1])
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1187
        while all_unversioned:
1188
            # yield any trailing unversioned paths
1189
            unversioned_path = all_unversioned.popleft()
1190
            to_kind, to_executable, to_stat = \
1191
                self.target._comparison_data(fake_entry, unversioned_path[1])
2255.7.96 by Robert Collins
Change _iter_changes interface to yield both old and new paths.
1192
            yield (None, (None, unversioned_path[1]), True, (False, False),
2255.7.85 by Robert Collins
Teach _iter_changes to gather unversioned path details upon request.
1193
                (None, None),
1194
                (None, unversioned_path[0][-1]),
1195
                (None, to_kind),
1196
                (None, to_executable))
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1197
        # Yield all remaining source paths
2012.1.7 by Aaron Bentley
Get tree._iter_changed down to ~ 1 stat per file
1198
        for path, from_entry in from_entries_by_dir:
2012.1.1 by Aaron Bentley
Implement change iterator
1199
            file_id = from_entry.file_id
1200
            if file_id in to_paths:
2255.7.96 by Robert Collins
Change _iter_changes interface to yield both old and new paths.
1201
                # already returned
2012.1.1 by Aaron Bentley
Implement change iterator
1202
                continue
5390.3.3 by Andrew Bennetts
Use has_id rather than __contains__; expand NEWS entry; add What's New entry.
1203
            if not self.target.has_id(file_id):
2255.7.96 by Robert Collins
Change _iter_changes interface to yield both old and new paths.
1204
                # common case - paths we have not emitted are not present in
1205
                # target.
1206
                to_path = None
1207
            else:
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1208
                to_path = self.target.id2path(file_id)
2012.1.16 by Aaron Bentley
Support progress bars in iter_changes
1209
            entry_count += 1
1210
            if pb is not None:
1211
                pb.update('comparing files', entry_count, num_entries)
2012.1.1 by Aaron Bentley
Implement change iterator
1212
            versioned = (True, False)
1213
            parent = (from_entry.parent_id, None)
1214
            name = (from_entry.name, None)
2012.1.7 by Aaron Bentley
Get tree._iter_changed down to ~ 1 stat per file
1215
            from_kind, from_executable, stat_value = \
2255.2.122 by Robert Collins
Alter intertree implementation tests to let dirstate inter-trees be correctly parameterised.
1216
                self.source._comparison_data(from_entry, path)
2012.1.7 by Aaron Bentley
Get tree._iter_changed down to ~ 1 stat per file
1217
            kind = (from_kind, None)
2012.1.2 by Aaron Bentley
reimplement compare_trees
1218
            executable = (from_executable, None)
3619.4.1 by Robert Collins
Improve tests for the behaviour of Tree.iter_changes for missing paths that are only present in one tree, and fix found bugs. (Robert Collins)
1219
            changed_content = from_kind is not None
2012.1.1 by Aaron Bentley
Implement change iterator
1220
            # the parent's path is necessarily known at this point.
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1221
            changed_file_ids.append(file_id)
2255.7.96 by Robert Collins
Change _iter_changes interface to yield both old and new paths.
1222
            yield(file_id, (path, to_path), changed_content, versioned, parent,
2012.1.1 by Aaron Bentley
Implement change iterator
1223
                  name, kind, executable)
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1224
        changed_file_ids = set(changed_file_ids)
1225
        if specific_file_ids is not None:
1226
            for result in self._handle_precise_ids(precise_file_ids,
1227
                changed_file_ids):
1228
                yield result
1229
1230
    def _get_entry(self, tree, file_id):
1231
        """Get an inventory entry from a tree, with missing entries as None.
1232
1233
        If the tree raises NotImplementedError on accessing .inventory, then
1234
        this is worked around using iter_entries_by_dir on just the file id
1235
        desired.
1236
1237
        :param tree: The tree to lookup the entry in.
1238
        :param file_id: The file_id to lookup.
1239
        """
1240
        try:
1241
            inventory = tree.inventory
1242
        except NotImplementedError:
1243
            # No inventory available.
1244
            try:
1245
                iterator = tree.iter_entries_by_dir(specific_file_ids=[file_id])
1246
                return iterator.next()[1]
1247
            except StopIteration:
1248
                return None
1249
        else:
1250
            try:
1251
                return inventory[file_id]
1252
            except errors.NoSuchId:
1253
                return None
1254
1255
    def _handle_precise_ids(self, precise_file_ids, changed_file_ids,
1256
        discarded_changes=None):
1257
        """Fill out a partial iter_changes to be consistent.
1258
1259
        :param precise_file_ids: The file ids of parents that were seen during
1260
            the iter_changes.
1261
        :param changed_file_ids: The file ids of already emitted items.
1262
        :param discarded_changes: An optional dict of precalculated
1263
            iter_changes items which the partial iter_changes had not output
1264
            but had calculated.
1265
        :return: A generator of iter_changes items to output.
1266
        """
1267
        # process parents of things that had changed under the users
1268
        # requested paths to prevent incorrect paths or parent ids which
1269
        # aren't in the tree.
1270
        while precise_file_ids:
1271
            precise_file_ids.discard(None)
1272
            # Don't emit file_ids twice
1273
            precise_file_ids.difference_update(changed_file_ids)
1274
            if not precise_file_ids:
1275
                break
1276
            # If the there was something at a given output path in source, we
1277
            # have to include the entry from source in the delta, or we would
1278
            # be putting this entry into a used path.
1279
            paths = []
1280
            for parent_id in precise_file_ids:
1281
                try:
1282
                    paths.append(self.target.id2path(parent_id))
1283
                except errors.NoSuchId:
1284
                    # This id has been dragged in from the source by delta
1285
                    # expansion and isn't present in target at all: we don't
1286
                    # need to check for path collisions on it.
1287
                    pass
1288
            for path in paths:
1289
                old_id = self.source.path2id(path)
1290
                precise_file_ids.add(old_id)
1291
            precise_file_ids.discard(None)
1292
            current_ids = precise_file_ids
1293
            precise_file_ids = set()
1294
            # We have to emit all of precise_file_ids that have been altered.
1295
            # We may have to output the children of some of those ids if any
1296
            # directories have stopped being directories.
1297
            for file_id in current_ids:
1298
                # Examine file_id
1299
                if discarded_changes:
1300
                    result = discarded_changes.get(file_id)
1301
                    old_entry = None
1302
                else:
1303
                    result = None
1304
                if result is None:
1305
                    old_entry = self._get_entry(self.source, file_id)
1306
                    new_entry = self._get_entry(self.target, file_id)
1307
                    result, changes = self._changes_from_entries(
1308
                        old_entry, new_entry)
1309
                else:
1310
                    changes = True
4570.2.5 by Robert Collins
Review feedback, including finding a bug with changes at the root.
1311
                # Get this parents parent to examine.
1312
                new_parent_id = result[4][1]
1313
                precise_file_ids.add(new_parent_id)
4570.2.3 by Robert Collins
Change the way iter_changes treats specific files to prevent InconsistentDeltas.
1314
                if changes:
1315
                    if (result[6][0] == 'directory' and
1316
                        result[6][1] != 'directory'):
1317
                        # This stopped being a directory, the old children have
1318
                        # to be included.
1319
                        if old_entry is None:
1320
                            # Reusing a discarded change.
1321
                            old_entry = self._get_entry(self.source, file_id)
1322
                        for child in old_entry.children.values():
1323
                            precise_file_ids.add(child.file_id)
1324
                    changed_file_ids.add(result[0])
1325
                    yield result
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1326
1327
5837.1.2 by Jelmer Vernooij
Register defaults as first optimiser for existing Inter objects.
1328
InterTree.register_optimiser(InterTree)
1329
1330
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1331
class MultiWalker(object):
1332
    """Walk multiple trees simultaneously, getting combined results."""
1333
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1334
    # Note: This could be written to not assume you can do out-of-order
1335
    #       lookups. Instead any nodes that don't match in all trees could be
1336
    #       marked as 'deferred', and then returned in the final cleanup loop.
1337
    #       For now, I think it is "nicer" to return things as close to the
1338
    #       "master_tree" order as we can.
1339
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1340
    def __init__(self, master_tree, other_trees):
1341
        """Create a new MultiWalker.
1342
1343
        All trees being walked must implement "iter_entries_by_dir()", such
1344
        that they yield (path, object) tuples, where that object will have a
1345
        '.file_id' member, that can be used to check equality.
1346
3514.3.11 by John Arbash Meinel
Cleanups suggested by Ian
1347
        :param master_tree: All trees will be 'slaved' to the master_tree such
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1348
            that nodes in master_tree will be used as 'first-pass' sync points.
1349
            Any nodes that aren't in master_tree will be merged in a second
1350
            pass.
1351
        :param other_trees: A list of other trees to walk simultaneously.
1352
        """
1353
        self._master_tree = master_tree
1354
        self._other_trees = other_trees
1355
3514.3.8 by John Arbash Meinel
Start trying to refactor the big function into smaller ones.
1356
        # Keep track of any nodes that were properly processed just out of
1357
        # order, that way we don't return them at the end, we don't have to
1358
        # track *all* processed file_ids, just the out-of-order ones
1359
        self._out_of_order_processed = set()
1360
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1361
    @staticmethod
1362
    def _step_one(iterator):
1363
        """Step an iter_entries_by_dir iterator.
1364
1365
        :return: (has_more, path, ie)
1366
            If has_more is False, path and ie will be None.
1367
        """
1368
        try:
1369
            path, ie = iterator.next()
1370
        except StopIteration:
1371
            return False, None, None
1372
        else:
1373
            return True, path, ie
1374
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1375
    @staticmethod
1376
    def _cmp_path_by_dirblock(path1, path2):
1377
        """Compare two paths based on what directory they are in.
1378
1379
        This generates a sort order, such that all children of a directory are
1380
        sorted together, and grandchildren are in the same order as the
1381
        children appear. But all grandchildren come after all children.
1382
1383
        :param path1: first path
1384
        :param path2: the second path
1385
        :return: negative number if ``path1`` comes first,
1386
            0 if paths are equal
1387
            and a positive number if ``path2`` sorts first
1388
        """
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1389
        # Shortcut this special case
1390
        if path1 == path2:
1391
            return 0
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1392
        # This is stolen from _dirstate_helpers_py.py, only switching it to
1393
        # Unicode objects. Consider using encode_utf8() and then using the
1394
        # optimized versions, or maybe writing optimized unicode versions.
1395
        if not isinstance(path1, unicode):
1396
            raise TypeError("'path1' must be a unicode string, not %s: %r"
1397
                            % (type(path1), path1))
1398
        if not isinstance(path2, unicode):
1399
            raise TypeError("'path2' must be a unicode string, not %s: %r"
1400
                            % (type(path2), path2))
3514.3.13 by John Arbash Meinel
One code path was using a tuple, another a plain path
1401
        return cmp(MultiWalker._path_to_key(path1),
1402
                   MultiWalker._path_to_key(path2))
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1403
1404
    @staticmethod
3514.3.13 by John Arbash Meinel
One code path was using a tuple, another a plain path
1405
    def _path_to_key(path):
3514.3.11 by John Arbash Meinel
Cleanups suggested by Ian
1406
        dirname, basename = osutils.split(path)
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1407
        return (dirname.split(u'/'), basename)
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1408
3514.3.8 by John Arbash Meinel
Start trying to refactor the big function into smaller ones.
1409
    def _lookup_by_file_id(self, extra_entries, other_tree, file_id):
1410
        """Lookup an inventory entry by file_id.
1411
1412
        This is called when an entry is missing in the normal order.
1413
        Generally this is because a file was either renamed, or it was
1414
        deleted/added. If the entry was found in the inventory and not in
1415
        extra_entries, it will be added to self._out_of_order_processed
1416
1417
        :param extra_entries: A dictionary of {file_id: (path, ie)}.  This
1418
            should be filled with entries that were found before they were
1419
            used. If file_id is present, it will be removed from the
1420
            dictionary.
1421
        :param other_tree: The Tree to search, in case we didn't find the entry
1422
            yet.
1423
        :param file_id: The file_id to look for
1424
        :return: (path, ie) if found or (None, None) if not present.
1425
        """
1426
        if file_id in extra_entries:
1427
            return extra_entries.pop(file_id)
1428
        # TODO: Is id2path better as the first call, or is
1429
        #       inventory[file_id] better as a first check?
1430
        try:
1431
            cur_path = other_tree.id2path(file_id)
1432
        except errors.NoSuchId:
1433
            cur_path = None
1434
        if cur_path is None:
1435
            return (None, None)
1436
        else:
1437
            self._out_of_order_processed.add(file_id)
1438
            cur_ie = other_tree.inventory[file_id]
1439
            return (cur_path, cur_ie)
1440
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1441
    def iter_all(self):
1442
        """Match up the values in the different trees."""
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1443
        for result in self._walk_master_tree():
1444
            yield result
1445
        self._finish_others()
1446
        for result in self._walk_others():
1447
            yield result
1448
1449
    def _walk_master_tree(self):
1450
        """First pass, walk all trees in lock-step.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
1451
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1452
        When we are done, all nodes in the master_tree will have been
1453
        processed. _other_walkers, _other_entries, and _others_extra will be
1454
        set on 'self' for future processing.
1455
        """
1456
        # This iterator has the most "inlining" done, because it tends to touch
1457
        # every file in the tree, while the others only hit nodes that don't
1458
        # match.
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1459
        master_iterator = self._master_tree.iter_entries_by_dir()
1460
1461
        other_walkers = [other.iter_entries_by_dir()
1462
                         for other in self._other_trees]
1463
        other_entries = [self._step_one(walker) for walker in other_walkers]
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1464
        # Track extra nodes in the other trees
1465
        others_extra = [{} for i in xrange(len(self._other_trees))]
1466
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1467
        master_has_more = True
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1468
        step_one = self._step_one
1469
        lookup_by_file_id = self._lookup_by_file_id
1470
        out_of_order_processed = self._out_of_order_processed
1471
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1472
        while master_has_more:
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1473
            (master_has_more, path, master_ie) = step_one(master_iterator)
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1474
            if not master_has_more:
1475
                break
1476
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1477
            file_id = master_ie.file_id
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1478
            other_values = []
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1479
            other_values_append = other_values.append
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1480
            next_other_entries = []
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1481
            next_other_entries_append = next_other_entries.append
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1482
            for idx, (other_has_more, other_path, other_ie) in enumerate(other_entries):
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1483
                if not other_has_more:
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1484
                    other_values_append(lookup_by_file_id(
3514.3.8 by John Arbash Meinel
Start trying to refactor the big function into smaller ones.
1485
                        others_extra[idx], self._other_trees[idx], file_id))
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1486
                    next_other_entries_append((False, None, None))
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1487
                elif file_id == other_ie.file_id:
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1488
                    # This is the critical code path, as most of the entries
1489
                    # should match between most trees.
1490
                    other_values_append((other_path, other_ie))
1491
                    next_other_entries_append(step_one(other_walkers[idx]))
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1492
                else:
1493
                    # This walker did not match, step it until it either
1494
                    # matches, or we know we are past the current walker.
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1495
                    other_walker = other_walkers[idx]
1496
                    other_extra = others_extra[idx]
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1497
                    while (other_has_more and
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1498
                           self._cmp_path_by_dirblock(other_path, path) < 0):
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1499
                        other_file_id = other_ie.file_id
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1500
                        if other_file_id not in out_of_order_processed:
1501
                            other_extra[other_file_id] = (other_path, other_ie)
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1502
                        other_has_more, other_path, other_ie = \
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1503
                            step_one(other_walker)
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1504
                    if other_has_more and other_ie.file_id == file_id:
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1505
                        # We ended up walking to this point, match and step
1506
                        # again
1507
                        other_values_append((other_path, other_ie))
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1508
                        other_has_more, other_path, other_ie = \
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1509
                            step_one(other_walker)
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1510
                    else:
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1511
                        # This record isn't in the normal order, see if it
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1512
                        # exists at all.
1513
                        other_values_append(lookup_by_file_id(
1514
                            other_extra, self._other_trees[idx], file_id))
1515
                    next_other_entries_append((other_has_more, other_path,
3514.3.2 by John Arbash Meinel
Handle the case when a record is missing in base
1516
                                               other_ie))
3514.3.1 by John Arbash Meinel
Start working on a special walker that can iterate several trees at once.
1517
            other_entries = next_other_entries
1518
1519
            # We've matched all the walkers, yield this datapoint
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1520
            yield path, file_id, master_ie, other_values
1521
        self._other_walkers = other_walkers
1522
        self._other_entries = other_entries
1523
        self._others_extra = others_extra
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1524
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1525
    def _finish_others(self):
1526
        """Finish walking the other iterators, so we get all entries."""
1527
        for idx, info in enumerate(self._other_entries):
1528
            other_extra = self._others_extra[idx]
1529
            (other_has_more, other_path, other_ie) = info
3514.3.4 by John Arbash Meinel
Handle more cases when the other tree has extra nodes.
1530
            while other_has_more:
1531
                other_file_id = other_ie.file_id
3514.3.8 by John Arbash Meinel
Start trying to refactor the big function into smaller ones.
1532
                if other_file_id not in self._out_of_order_processed:
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1533
                    other_extra[other_file_id] = (other_path, other_ie)
3514.3.4 by John Arbash Meinel
Handle more cases when the other tree has extra nodes.
1534
                other_has_more, other_path, other_ie = \
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1535
                    self._step_one(self._other_walkers[idx])
1536
        del self._other_entries
1537
1538
    def _walk_others(self):
1539
        """Finish up by walking all the 'deferred' nodes."""
1540
        # TODO: One alternative would be to grab all possible unprocessed
1541
        #       file_ids, and then sort by path, and then yield them. That
1542
        #       might ensure better ordering, in case a caller strictly
1543
        #       requires parents before children.
1544
        for idx, other_extra in enumerate(self._others_extra):
3514.3.13 by John Arbash Meinel
One code path was using a tuple, another a plain path
1545
            others = sorted(other_extra.itervalues(),
1546
                            key=lambda x: self._path_to_key(x[0]))
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1547
            for other_path, other_ie in others:
1548
                file_id = other_ie.file_id
3514.3.7 by John Arbash Meinel
comment
1549
                # We don't need to check out_of_order_processed here, because
1550
                # the lookup_by_file_id will be removing anything processed
1551
                # from the extras cache
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1552
                other_extra.pop(file_id)
3514.3.5 by John Arbash Meinel
Handle some edge cases when we have multiple other trees.
1553
                other_values = [(None, None) for i in xrange(idx)]
1554
                other_values.append((other_path, other_ie))
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1555
                for alt_idx, alt_extra in enumerate(self._others_extra[idx+1:]):
3514.3.8 by John Arbash Meinel
Start trying to refactor the big function into smaller ones.
1556
                    alt_idx = alt_idx + idx + 1
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1557
                    alt_extra = self._others_extra[alt_idx]
1558
                    alt_tree = self._other_trees[alt_idx]
3514.3.8 by John Arbash Meinel
Start trying to refactor the big function into smaller ones.
1559
                    other_values.append(self._lookup_by_file_id(
3514.3.9 by John Arbash Meinel
Refactor the large function into multiple small ones.
1560
                                            alt_extra, alt_tree, file_id))
3514.3.3 by John Arbash Meinel
Handle when the other tree has extra nodes, and we need to yield them.
1561
                yield other_path, file_id, None, other_values