/brz/remove-bazaar

To get this branch, use:
bzr branch http://gegoxaren.bato24.eu/bzr/brz/remove-bazaar
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
1
# Copyright (C) 2011 Canonical Ltd
2
#
5820.1.18 by INADA Naoki
Add copyright for some function.
3
# UTextWrapper._handle_long_word, UTextWrapper._wrap_chunks,
5820.1.27 by INADA Naoki
Fix error when fix_sentence_endings=True.
4
# UTextWrapper._fix_sentence_endings, wrap and fill is copied from Python's
5
# textwrap module (under PSF license) and modified for support CJK.
5820.1.18 by INADA Naoki
Add copyright for some function.
6
# Original Copyright for these functions:
7
#
8
# Copyright (C) 1999-2001 Gregory P. Ward.
9
# Copyright (C) 2002, 2003 Python Software Foundation.
10
#
11
# Written by Greg Ward <gward@python.net>
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
12
# This program is free software; you can redistribute it and/or modify
13
# it under the terms of the GNU General Public License as published by
14
# the Free Software Foundation; either version 2 of the License, or
15
# (at your option) any later version.
16
#
17
# This program is distributed in the hope that it will be useful,
18
# but WITHOUT ANY WARRANTY; without even the implied warranty of
19
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
20
# GNU General Public License for more details.
21
#
22
# You should have received a copy of the GNU General Public License
23
# along with this program; if not, write to the Free Software
24
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
25
6379.6.3 by Jelmer Vernooij
Use absolute_import.
26
from __future__ import absolute_import
27
5820.1.15 by Martin
Cope with lack of TextWrapper.drop_whitespace before Python 2.6
28
import sys
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
29
import textwrap
30
from unicodedata import east_asian_width as _eawidth
31
6624 by Jelmer Vernooij
Merge Python3 porting work ('py3 pokes')
32
from . import osutils
5820.1.10 by INADA Naoki
utextwrap: Change a way to split between CJK characters.
33
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
34
__all__ = ["UTextWrapper", "fill", "wrap"]
35
36
class UTextWrapper(textwrap.TextWrapper):
37
    """
38
    Extend TextWrapper for Unicode.
39
40
    This textwrapper handles east asian double width and split word
41
    even if !break_long_words when word contains double width
42
    characters.
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
43
44
    :param ambiguous_width: (keyword argument) width for character when
45
                            unicodedata.east_asian_width(c) == 'A'
5820.1.21 by INADA Naoki
Change default value of ambiguous_width from 2 to 1.
46
                            (default: 1)
5820.1.22 by INADA Naoki
Add document of some limitations in docstring.
47
48
    Limitations:
49
    * expand_tabs doesn't fixed. It uses len() for calculating width
50
      of string on left of TAB.
51
    * Handles one codeunit as a single character having 1 or 2 width.
52
      This is not correct when there are surrogate pairs, combined
53
      characters or zero-width characters.
54
    * Treats all asian character are line breakable. But it is not
55
      true because line breaking is prohibited around some characters.
56
      (For example, breaking before punctation mark is prohibited.)
57
      See UAX # 14 "UNICODE LINE BREAKING ALGORITHM"
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
58
    """
5820.1.15 by Martin
Cope with lack of TextWrapper.drop_whitespace before Python 2.6
59
5820.1.9 by INADA Naoki
Default width of UTextWrapper is also osutils.terminal_widtth() and
60
    def __init__(self, width=None, **kwargs):
61
        if width is None:
62
            width = (osutils.terminal_width() or
63
                        osutils.default_terminal_width) - 1
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
64
5820.1.21 by INADA Naoki
Change default value of ambiguous_width from 2 to 1.
65
        ambi_width = kwargs.pop('ambiguous_width', 1)
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
66
        if ambi_width == 1:
67
            self._east_asian_doublewidth = 'FW'
68
        elif ambi_width == 2:
69
            self._east_asian_doublewidth = 'FWA'
70
        else:
71
            raise ValueError("ambiguous_width should be 1 or 2")
72
5820.1.9 by INADA Naoki
Default width of UTextWrapper is also osutils.terminal_widtth() and
73
        textwrap.TextWrapper.__init__(self, width, **kwargs)
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
74
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
75
    def _unicode_char_width(self, uc):
76
        """Return width of character `uc`.
77
78
        :param:     uc      Single unicode character.
79
        """
80
        # 'A' means width of the character is not be able to determine.
81
        # We assume that it's width is 2 because longer wrap may over
82
        # terminal width but shorter wrap may be acceptable.
83
        return (_eawidth(uc) in self._east_asian_doublewidth and 2) or 1
84
85
    def _width(self, s):
86
        """Returns width for s.
5820.1.26 by INADA Naoki
Cleanup. Remove spaces in empty line and shorten line having 80 characters.
87
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
88
        When s is unicode, take care of east asian width.
89
        When s is bytes, treat all byte is single width character.
90
        """
91
        charwidth = self._unicode_char_width
92
        return sum(charwidth(c) for c in s)
93
94
    def _cut(self, s, width):
95
        """Returns head and rest of s. (head+rest == s)
96
97
        Head is large as long as _width(head) <= width.
98
        """
99
        w = 0
100
        charwidth = self._unicode_char_width
101
        for pos, c in enumerate(s):
102
            w += charwidth(c)
103
            if w > width:
104
                return s[:pos], s[pos:]
105
        return s, u''
106
5820.1.27 by INADA Naoki
Fix error when fix_sentence_endings=True.
107
    def _fix_sentence_endings(self, chunks):
108
        """_fix_sentence_endings(chunks : [string])
109
110
        Correct for sentence endings buried in 'chunks'.  Eg. when the
111
        original text contains "... foo.\nBar ...", munge_whitespace()
112
        and split() will convert that to [..., "foo.", " ", "Bar", ...]
113
        which has one too few spaces; this method simply changes the one
114
        space to two.
115
116
        Note: This function is copied from textwrap.TextWrap and modified
117
        to use unicode always.
118
        """
119
        i = 0
120
        L = len(chunks)-1
121
        patsearch = self.sentence_end_re.search
122
        while i < L:
123
            if chunks[i+1] == u" " and patsearch(chunks[i]):
124
                chunks[i+1] = u"  "
125
                i += 2
126
            else:
127
                i += 1
128
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
129
    def _handle_long_word(self, chunks, cur_line, cur_len, width):
5820.1.10 by INADA Naoki
utextwrap: Change a way to split between CJK characters.
130
        # Figure out when indent is larger than the specified width, and make
131
        # sure at least one character is stripped off on every pass
132
        if width < 2:
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
133
            space_left = chunks[-1] and self._width(chunks[-1][0]) or 1
5820.1.10 by INADA Naoki
utextwrap: Change a way to split between CJK characters.
134
        else:
135
            space_left = width - cur_len
136
137
        # If we're allowed to break long words, then do so: put as much
138
        # of the next chunk onto the current line as will fit.
139
        if self.break_long_words:
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
140
            head, rest = self._cut(chunks[-1], space_left)
5820.1.10 by INADA Naoki
utextwrap: Change a way to split between CJK characters.
141
            cur_line.append(head)
5820.1.5 by INADA Naoki
Make UTextWrapper support byte string and add tests including Python's
142
            if rest:
5820.1.10 by INADA Naoki
utextwrap: Change a way to split between CJK characters.
143
                chunks[-1] = rest
144
            else:
145
                del chunks[-1]
146
147
        # Otherwise, we have to preserve the long word intact.  Only add
148
        # it to the current line if there's nothing already there --
149
        # that minimizes how much we violate the width constraint.
150
        elif not cur_line:
151
            cur_line.append(chunks.pop())
152
153
        # If we're not allowed to break long words, and there's already
154
        # text on the current line, do nothing.  Next time through the
155
        # main loop of _wrap_chunks(), we'll wind up here again, but
156
        # cur_len will be zero, so the next line will be entirely
157
        # devoted to the long word that we can't handle right now.
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
158
159
    def _wrap_chunks(self, chunks):
160
        lines = []
161
        if self.width <= 0:
162
            raise ValueError("invalid width %r (must be > 0)" % self.width)
163
164
        # Arrange in reverse order so items can be efficiently popped
165
        # from a stack of chucks.
166
        chunks.reverse()
167
168
        while chunks:
169
170
            # Start the list of chunks that will make up the current line.
171
            # cur_len is just the length of all the chunks in cur_line.
172
            cur_line = []
173
            cur_len = 0
174
175
            # Figure out which static string will prefix this line.
176
            if lines:
177
                indent = self.subsequent_indent
178
            else:
179
                indent = self.initial_indent
180
181
            # Maximum width for this line.
182
            width = self.width - len(indent)
183
184
            # First chunk on line is whitespace -- drop it, unless this
185
            # is the very beginning of the text (ie. no lines started yet).
186
            if self.drop_whitespace and chunks[-1].strip() == '' and lines:
187
                del chunks[-1]
188
189
            while chunks:
190
                # Use _width instead of len for east asian width
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
191
                l = self._width(chunks[-1])
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
192
193
                # Can at least squeeze this chunk onto the current line.
194
                if cur_len + l <= width:
195
                    cur_line.append(chunks.pop())
196
                    cur_len += l
197
198
                # Nope, this line is full.
199
                else:
200
                    break
201
202
            # The current line is full, and the next chunk is too big to
203
            # fit on *any* line (not just this one).
5820.1.19 by INADA Naoki
Add keyword parameter 'ambiguous_width' that specifies width for character
204
            if chunks and self._width(chunks[-1]) > width:
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
205
                self._handle_long_word(chunks, cur_line, cur_len, width)
206
207
            # If the last chunk on this line is all whitespace, drop it.
5820.1.26 by INADA Naoki
Cleanup. Remove spaces in empty line and shorten line having 80 characters.
208
            if self.drop_whitespace and cur_line and not cur_line[-1].strip():
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
209
                del cur_line[-1]
210
211
            # Convert current line back to a string and store it in list
212
            # of all lines (return value).
213
            if cur_line:
5820.1.27 by INADA Naoki
Fix error when fix_sentence_endings=True.
214
                lines.append(indent + u''.join(cur_line))
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
215
216
        return lines
217
5820.1.10 by INADA Naoki
utextwrap: Change a way to split between CJK characters.
218
    def _split(self, text):
7012.2.1 by Jelmer Vernooij
Always pass in unicode to utextwrap.
219
        chunks = textwrap.TextWrapper._split(self, osutils.safe_unicode(text))
5820.1.10 by INADA Naoki
utextwrap: Change a way to split between CJK characters.
220
        cjk_split_chunks = []
221
        for chunk in chunks:
222
            prev_pos = 0
223
            for pos, char in enumerate(chunk):
5820.1.27 by INADA Naoki
Fix error when fix_sentence_endings=True.
224
                if self._unicode_char_width(char) == 2:
5820.1.10 by INADA Naoki
utextwrap: Change a way to split between CJK characters.
225
                    if prev_pos < pos:
226
                        cjk_split_chunks.append(chunk[prev_pos:pos])
227
                    cjk_split_chunks.append(char)
228
                    prev_pos = pos+1
229
            if prev_pos < len(chunk):
230
                cjk_split_chunks.append(chunk[prev_pos:])
231
        return cjk_split_chunks
232
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
233
    def wrap(self, text):
234
        # ensure text is unicode
7012.2.1 by Jelmer Vernooij
Always pass in unicode to utextwrap.
235
        return textwrap.TextWrapper.wrap(self, osutils.safe_unicode(text))
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
236
237
# -- Convenience interface ---------------------------------------------
238
5820.1.2 by INADA Naoki
bzrlib.utextwrap uses bzrlib.osutils.terminal_width() when width is not specified.
239
def wrap(text, width=None, **kwargs):
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
240
    """Wrap a single paragraph of text, returning a list of wrapped lines.
241
242
    Reformat the single paragraph in 'text' so it fits in lines of no
243
    more than 'width' columns, and return a list of wrapped lines.  By
244
    default, tabs in 'text' are expanded with string.expandtabs(), and
245
    all other whitespace characters (including newline) are converted to
246
    space.  See TextWrapper class for available keyword args to customize
247
    wrapping behaviour.
248
    """
5820.1.9 by INADA Naoki
Default width of UTextWrapper is also osutils.terminal_widtth() and
249
    return UTextWrapper(width=width, **kwargs).wrap(text)
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
250
5820.1.2 by INADA Naoki
bzrlib.utextwrap uses bzrlib.osutils.terminal_width() when width is not specified.
251
def fill(text, width=None, **kwargs):
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
252
    """Fill a single paragraph of text, returning a new string.
253
254
    Reformat the single paragraph in 'text' to fit in lines of no more
255
    than 'width' columns, and return a new string containing the entire
256
    wrapped paragraph.  As with wrap(), tabs are expanded and other
257
    whitespace characters converted to space.  See TextWrapper class for
258
    available keyword args to customize wrapping behaviour.
259
    """
5820.1.9 by INADA Naoki
Default width of UTextWrapper is also osutils.terminal_widtth() and
260
    return UTextWrapper(width=width, **kwargs).fill(text)
5820.1.1 by INADA Naoki
Add utextwrap that is same to textwrap but supports double width characters in east asia.
261