/brz/remove-bazaar

To get this branch, use:
bzr branch http://gegoxaren.bato24.eu/bzr/brz/remove-bazaar
4763.2.4 by John Arbash Meinel
merge bzr.2.1 in preparation for NEWS entry.
1
# Copyright (C) 2006-2010 Canonical Ltd
2018.5.26 by Andrew Bennetts
Extract a simple SmartClient class from RemoteTransport, and a hack to avoid VFS operations when probing for a bzrdir over a smart transport.
2
#
3
# This program is free software; you can redistribute it and/or modify
4
# it under the terms of the GNU General Public License as published by
5
# the Free Software Foundation; either version 2 of the License, or
6
# (at your option) any later version.
7
#
8
# This program is distributed in the hope that it will be useful,
9
# but WITHOUT ANY WARRANTY; without even the implied warranty of
10
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11
# GNU General Public License for more details.
12
#
13
# You should have received a copy of the GNU General Public License
14
# along with this program; if not, write to the Free Software
4183.7.1 by Sabin Iacob
update FSF mailing address
15
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
2018.5.26 by Andrew Bennetts
Extract a simple SmartClient class from RemoteTransport, and a hack to avoid VFS operations when probing for a bzrdir over a smart transport.
16
4797.85.35 by John Arbash Meinel
Do a lookup in 'request.request_handlers', to figure out if a request can be retried.
17
from bzrlib import lazy_import
18
lazy_import.lazy_import(globals(), """
19
from bzrlib.smart import request as _mod_request
20
""")
21
3245.4.42 by Andrew Bennetts
Make _SmartClient automatically detect and use the highest protocol version compatible with the server.
22
import bzrlib
3195.3.17 by Andrew Bennetts
Some tests now passing using protocol 3.
23
from bzrlib.smart import message, protocol
3703.3.3 by Andrew Bennetts
Neater effort tests for push by using a _SmartClient.hooks['call'] hook.
24
from bzrlib import (
4797.91.5 by John Arbash Meinel
Add -Dnoretry as a way to disable all retrying code.
25
    debug,
3703.3.3 by Andrew Bennetts
Neater effort tests for push by using a _SmartClient.hooks['call'] hook.
26
    errors,
27
    hooks,
4797.85.18 by John Arbash Meinel
Start trying to reconnect to the server if we get a ConnectionReset during the transmission.
28
    trace,
3703.3.3 by Andrew Bennetts
Neater effort tests for push by using a _SmartClient.hooks['call'] hook.
29
    )
2018.5.26 by Andrew Bennetts
Extract a simple SmartClient class from RemoteTransport, and a hack to avoid VFS operations when probing for a bzrdir over a smart transport.
30
31
2414.1.4 by Andrew Bennetts
Rename SmartClient to _SmartClient.
32
class _SmartClient(object):
2018.5.26 by Andrew Bennetts
Extract a simple SmartClient class from RemoteTransport, and a hack to avoid VFS operations when probing for a bzrdir over a smart transport.
33
3431.3.2 by Andrew Bennetts
Remove 'base' from _SmartClient entirely, now that the medium has it.
34
    def __init__(self, medium, headers=None):
3104.4.4 by Andrew Bennetts
Rename parameter to _SmartClient.__init__ to be less confusing.
35
        """Constructor.
36
3313.2.1 by Andrew Bennetts
Change _SmartClient's API to accept a medium and a base, rather than a _SharedConnection.
37
        :param medium: a SmartClientMedium
3104.4.4 by Andrew Bennetts
Rename parameter to _SmartClient.__init__ to be less confusing.
38
        """
3313.2.1 by Andrew Bennetts
Change _SmartClient's API to accept a medium and a base, rather than a _SharedConnection.
39
        self._medium = medium
3245.4.42 by Andrew Bennetts
Make _SmartClient automatically detect and use the highest protocol version compatible with the server.
40
        if headers is None:
41
            self._headers = {'Software version': bzrlib.__version__}
42
        else:
43
            self._headers = dict(headers)
44
4964.2.2 by Martin Pool
Add SmartClient repr
45
    def __repr__(self):
46
        return '%s(%r)' % (self.__class__.__name__, self._medium)
47
3245.4.42 by Andrew Bennetts
Make _SmartClient automatically detect and use the highest protocol version compatible with the server.
48
    def _call_and_read_response(self, method, args, body=None, readv_body=None,
3842.3.4 by Andrew Bennetts
TestStacking.test_fetch_copies_from_stacked_on now passes using the VersionedFile.insert_record_stream RPC; lots of debugging cruft needs removal though.
49
            body_stream=None, expect_response_body=True):
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
50
        request = _SmartClientRequest(self, method, args, body=body,
51
            readv_body=readv_body, body_stream=body_stream,
52
            expect_response_body=expect_response_body)
53
        return request.call_and_read_response()
3245.4.42 by Andrew Bennetts
Make _SmartClient automatically detect and use the highest protocol version compatible with the server.
54
2018.5.26 by Andrew Bennetts
Extract a simple SmartClient class from RemoteTransport, and a hack to avoid VFS operations when probing for a bzrdir over a smart transport.
55
    def call(self, method, *args):
56
        """Call a method on the remote server."""
2414.1.2 by Andrew Bennetts
Deal with review comments.
57
        result, protocol = self.call_expecting_body(method, *args)
2414.1.1 by Andrew Bennetts
Add some unicode-related tests from the hpss branch, and a few other nits (also from the hpss branch).
58
        protocol.cancel_read_body()
59
        return result
60
2414.1.2 by Andrew Bennetts
Deal with review comments.
61
    def call_expecting_body(self, method, *args):
62
        """Call a method and return the result and the protocol object.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
63
2414.1.2 by Andrew Bennetts
Deal with review comments.
64
        The body can be read like so::
65
66
            result, smart_protocol = smart_client.call_expecting_body(...)
67
            body = smart_protocol.read_body_bytes()
68
        """
3245.4.42 by Andrew Bennetts
Make _SmartClient automatically detect and use the highest protocol version compatible with the server.
69
        return self._call_and_read_response(
70
            method, args, expect_response_body=True)
2018.5.26 by Andrew Bennetts
Extract a simple SmartClient class from RemoteTransport, and a hack to avoid VFS operations when probing for a bzrdir over a smart transport.
71
72
    def call_with_body_bytes(self, method, args, body):
73
        """Call a method on the remote server with body bytes."""
2018.5.131 by Andrew Bennetts
Be strict about unicode passed to transport.put_{bytes,file} and SmartClient.call_with_body_bytes, fixing part of TestLockableFiles_RemoteLockDir.test_read_write.
74
        if type(method) is not str:
75
            raise TypeError('method must be a byte string, not %r' % (method,))
76
        for arg in args:
77
            if type(arg) is not str:
78
                raise TypeError('args must be byte strings, not %r' % (args,))
79
        if type(body) is not str:
80
            raise TypeError('body must be byte string, not %r' % (body,))
3245.4.42 by Andrew Bennetts
Make _SmartClient automatically detect and use the highest protocol version compatible with the server.
81
        response, response_handler = self._call_and_read_response(
82
            method, args, body=body, expect_response_body=False)
83
        return response
2414.1.1 by Andrew Bennetts
Add some unicode-related tests from the hpss branch, and a few other nits (also from the hpss branch).
84
3184.1.10 by Robert Collins
Change the smart server verb for Repository.stream_revisions_chunked to use SearchResults as the request mechanism for downloads.
85
    def call_with_body_bytes_expecting_body(self, method, args, body):
86
        """Call a method on the remote server with body bytes."""
87
        if type(method) is not str:
88
            raise TypeError('method must be a byte string, not %r' % (method,))
3842.3.2 by Andrew Bennetts
Revert the RemoteVersionedFiles.get_parent_map implementation, leaving just the skeleton of RemoteVersionedFiles.
89
        for arg in args:
90
            if type(arg) is not str:
91
                raise TypeError('args must be byte strings, not %r' % (args,))
3184.1.10 by Robert Collins
Change the smart server verb for Repository.stream_revisions_chunked to use SearchResults as the request mechanism for downloads.
92
        if type(body) is not str:
93
            raise TypeError('body must be byte string, not %r' % (body,))
3245.4.42 by Andrew Bennetts
Make _SmartClient automatically detect and use the highest protocol version compatible with the server.
94
        response, response_handler = self._call_and_read_response(
95
            method, args, body=body, expect_response_body=True)
96
        return (response, response_handler)
3184.1.10 by Robert Collins
Change the smart server verb for Repository.stream_revisions_chunked to use SearchResults as the request mechanism for downloads.
97
3245.4.24 by Andrew Bennetts
Consistently raise errors from the server as ErrorFromSmartServer exceptions.
98
    def call_with_body_readv_array(self, args, body):
3245.4.42 by Andrew Bennetts
Make _SmartClient automatically detect and use the highest protocol version compatible with the server.
99
        response, response_handler = self._call_and_read_response(
100
                args[0], args[1:], readv_body=body, expect_response_body=True)
101
        return (response, response_handler)
3245.4.24 by Andrew Bennetts
Consistently raise errors from the server as ErrorFromSmartServer exceptions.
102
3842.3.4 by Andrew Bennetts
TestStacking.test_fetch_copies_from_stacked_on now passes using the VersionedFile.insert_record_stream RPC; lots of debugging cruft needs removal though.
103
    def call_with_body_stream(self, args, stream):
104
        response, response_handler = self._call_and_read_response(
105
                args[0], args[1:], body_stream=stream,
106
                expect_response_body=False)
107
        return (response, response_handler)
108
2414.1.1 by Andrew Bennetts
Add some unicode-related tests from the hpss branch, and a few other nits (also from the hpss branch).
109
    def remote_path_from_transport(self, transport):
2414.1.2 by Andrew Bennetts
Deal with review comments.
110
        """Convert transport into a path suitable for using in a request.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
111
2414.1.2 by Andrew Bennetts
Deal with review comments.
112
        Note that the resulting remote path doesn't encode the host name or
113
        anything but path, so it is only safe to use it in requests sent over
114
        the medium from the matching transport.
115
        """
3431.3.11 by Andrew Bennetts
Push remote_path_from_transport logic into SmartClientMedium, removing special-casing of bzr+http from _SmartClient.
116
        return self._medium.remote_path_from_transport(transport)
3297.3.1 by Andrew Bennetts
Raise UnknownSmartMethod automatically from read_response_tuple.
117
3703.3.3 by Andrew Bennetts
Neater effort tests for push by using a _SmartClient.hooks['call'] hook.
118
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
119
class _SmartClientRequest(object):
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
120
    """Encapsulate the logic for a single request.
121
122
    This class handles things like reconnecting and sending the request a
123
    second time when the connection is reset in the middle. It also handles the
124
    multiple requests that get made if we don't know what protocol the server
125
    supports yet.
126
127
    Generally, you build up one of these objects, passing in the arguments that
128
    you want to send to the server, and then use 'call_and_read_response' to
129
    get the response from the server.
130
    """
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
131
132
    def __init__(self, client, method, args, body=None, readv_body=None,
133
                 body_stream=None, expect_response_body=True):
134
        self.client = client
135
        self.method = method
136
        self.args = args
137
        self.body = body
138
        self.readv_body = readv_body
139
        self.body_stream = body_stream
140
        self.expect_response_body = expect_response_body
141
142
    def call_and_read_response(self):
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
143
        """Send the request to the server, and read the initial response.
144
145
        This doesn't read all of the body content of the response, instead it
146
        returns (response_tuple, response_handler). response_tuple is the 'ok',
147
        or 'error' information, and 'response_handler' can be used to get the
148
        content stream out.
149
        """
4797.85.29 by John Arbash Meinel
Move the hooks code over as well.
150
        self._run_call_hooks()
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
151
        protocol_version = self.client._medium._protocol_version
152
        if protocol_version is None:
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
153
            return self._call_determining_protocol_version()
154
        else:
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
155
            return self._call(protocol_version)
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
156
4797.85.35 by John Arbash Meinel
Do a lookup in 'request.request_handlers', to figure out if a request can be retried.
157
    def _is_safe_to_send_twice(self):
158
        """Check if the current method is re-entrant safe."""
4797.85.37 by John Arbash Meinel
We should refuse to retry on read for -Dnoretry as well.
159
        if self.body_stream is not None or 'noretry' in debug.debug_flags:
4797.85.35 by John Arbash Meinel
Do a lookup in 'request.request_handlers', to figure out if a request can be retried.
160
            # We can't restart a body stream that has already been consumed.
161
            return False
162
        request_type = _mod_request.request_handlers.get_info(self.method)
163
        if request_type in ('read', 'idem', 'semi'):
164
            return True
165
        # If we have gotten this far, 'stream' cannot be retried, because we
166
        # already consumed the local stream.
167
        if request_type in ('semivfs', 'mutate', 'stream'):
168
            return False
169
        trace.mutter('Unknown request type: %s for method %s'
170
                     % (request_type, self.method))
171
        return False
172
4797.85.29 by John Arbash Meinel
Move the hooks code over as well.
173
    def _run_call_hooks(self):
174
        if not _SmartClient.hooks['call']:
175
            return
176
        params = CallHookParams(self.method, self.args, self.body,
177
                                self.readv_body, self.client._medium)
178
        for hook in _SmartClient.hooks['call']:
179
            hook(params)
180
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
181
    def _call(self, protocol_version):
182
        """We know the protocol version.
4797.85.29 by John Arbash Meinel
Move the hooks code over as well.
183
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
184
        So this just sends the request, and then reads the response. This is
185
        where the code will be to retry requests if the connection is closed.
186
        """
187
        response_handler = self._send(protocol_version)
4797.85.34 by John Arbash Meinel
Start implementing retry during read.
188
        try:
189
            response_tuple = response_handler.read_response_tuple(
190
                expect_body=self.expect_response_body)
191
        except errors.ConnectionReset, e:
192
            self.client._medium.reset()
4797.85.35 by John Arbash Meinel
Do a lookup in 'request.request_handlers', to figure out if a request can be retried.
193
            if not self._is_safe_to_send_twice():
194
                raise
4797.85.34 by John Arbash Meinel
Start implementing retry during read.
195
            trace.warning('ConnectionReset reading response for %r, retrying'
196
                          % (self.method,))
197
            trace.log_exception_quietly()
4797.85.38 by John Arbash Meinel
Use the raw _send_no_retry when retrying from a read.
198
            encoder, response_handler = self._construct_protocol(
199
                protocol_version)
200
            self._send_no_retry(encoder)
4797.85.34 by John Arbash Meinel
Start implementing retry during read.
201
            response_tuple = response_handler.read_response_tuple(
202
                expect_body=self.expect_response_body)
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
203
        return (response_tuple, response_handler)
204
205
    def _call_determining_protocol_version(self):
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
206
        """Determine what protocol the remote server supports.
207
208
        We do this by placing a request in the most recent protocol, and
209
        handling the UnexpectedProtocolVersionMarker from the server.
210
        """
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
211
        for protocol_version in [3, 2]:
212
            if protocol_version == 2:
213
                # If v3 doesn't work, the remote side is older than 1.6.
214
                self.client._medium._remember_remote_is_before((1, 6))
215
            try:
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
216
                response_tuple, response_handler = self._call(protocol_version)
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
217
            except errors.UnexpectedProtocolVersionMarker, err:
218
                # TODO: We could recover from this without disconnecting if
219
                # we recognise the protocol version.
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
220
                trace.warning(
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
221
                    'Server does not understand Bazaar network protocol %d,'
222
                    ' reconnecting.  (Upgrade the server to avoid this.)'
223
                    % (protocol_version,))
224
                self.client._medium.disconnect()
225
                continue
226
            except errors.ErrorFromSmartServer:
227
                # If we received an error reply from the server, then it
228
                # must be ok with this protocol version.
229
                self.client._medium._protocol_version = protocol_version
230
                raise
231
            else:
232
                self.client._medium._protocol_version = protocol_version
233
                return response_tuple, response_handler
234
        raise errors.SmartProtocolError(
235
            'Server is not a Bazaar server: ' + str(err))
236
4797.85.28 by John Arbash Meinel
Move all of the send logic into the _SmartClientRequest class.
237
    def _construct_protocol(self, version):
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
238
        """Build the encoding stack for a given protocol version."""
4797.85.28 by John Arbash Meinel
Move all of the send logic into the _SmartClientRequest class.
239
        request = self.client._medium.get_request()
240
        if version == 3:
241
            request_encoder = protocol.ProtocolThreeRequester(request)
242
            response_handler = message.ConventionalResponseHandler()
243
            response_proto = protocol.ProtocolThreeDecoder(
244
                response_handler, expect_version_marker=True)
245
            response_handler.setProtoAndMediumRequest(response_proto, request)
246
        elif version == 2:
247
            request_encoder = protocol.SmartClientRequestProtocolTwo(request)
248
            response_handler = request_encoder
249
        else:
250
            request_encoder = protocol.SmartClientRequestProtocolOne(request)
251
            response_handler = request_encoder
252
        return request_encoder, response_handler
253
254
    def _send(self, protocol_version):
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
255
        """Encode the request, and send it to the server.
256
257
        This will retry a request if we get a ConnectionReset while sending the
258
        request to the server. (Unless we have a body_stream that we have
259
        already started consuming, since we can't restart body_streams)
260
261
        :return: response_handler as defined by _construct_protocol
262
        """
4797.85.28 by John Arbash Meinel
Move all of the send logic into the _SmartClientRequest class.
263
        encoder, response_handler = self._construct_protocol(protocol_version)
264
        try:
265
            self._send_no_retry(encoder)
266
        except errors.ConnectionReset, e:
267
            # If we fail during the _send_no_retry phase, then we can
268
            # be confident that the server did not get our request, because we
269
            # haven't started waiting for the reply yet. So try the request
270
            # again. We only issue a single retry, because if the connection
271
            # really is down, there is no reason to loop endlessly.
272
273
            # Connection is dead, so close our end of it.
274
            self.client._medium.reset()
4797.91.5 by John Arbash Meinel
Add -Dnoretry as a way to disable all retrying code.
275
            if (('noretry' in debug.debug_flags)
4797.93.4 by John Arbash Meinel
Merge in the debug flag updates.
276
                or (self.body_stream is not None
277
                    and encoder.body_stream_started)):
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
278
                # We can't restart a body_stream that has been partially
279
                # consumed, so we don't retry.
4797.93.2 by John Arbash Meinel
Merge the refactored SmartClientRequest code, and update to match it.
280
                # Note: We don't have to worry about
281
                #   SmartClientRequestProtocolOne or Two, because they don't
282
                #   support client-side body streams.
4797.85.28 by John Arbash Meinel
Move all of the send logic into the _SmartClientRequest class.
283
                raise
4797.91.4 by John Arbash Meinel
Clean up the logging slightly.
284
            trace.warning('ConnectionReset calling %r, retrying'
285
                          % (self.method,))
4797.85.28 by John Arbash Meinel
Move all of the send logic into the _SmartClientRequest class.
286
            trace.log_exception_quietly()
287
            encoder, response_handler = self._construct_protocol(
288
                protocol_version)
289
            self._send_no_retry(encoder)
290
        return response_handler
291
292
    def _send_no_retry(self, encoder):
4797.85.30 by John Arbash Meinel
Document the functions in the new class.
293
        """Just encode the request and try to send it."""
4797.85.28 by John Arbash Meinel
Move all of the send logic into the _SmartClientRequest class.
294
        encoder.set_headers(self.client._headers)
295
        if self.body is not None:
296
            if self.readv_body is not None:
297
                raise AssertionError(
298
                    "body and readv_body are mutually exclusive.")
299
            if self.body_stream is not None:
300
                raise AssertionError(
301
                    "body and body_stream are mutually exclusive.")
302
            encoder.call_with_body_bytes((self.method, ) + self.args, self.body)
303
        elif self.readv_body is not None:
304
            if self.body_stream is not None:
305
                raise AssertionError(
306
                    "readv_body and body_stream are mutually exclusive.")
307
            encoder.call_with_body_readv_array((self.method, ) + self.args,
308
                                               self.readv_body)
309
        elif self.body_stream is not None:
310
            encoder.call_with_body_stream((self.method, ) + self.args,
311
                                          self.body_stream)
312
        else:
313
            encoder.call(self.method, *self.args)
314
4797.85.27 by John Arbash Meinel
I've started passing around too many parameters to too many functions.
315
3703.3.3 by Andrew Bennetts
Neater effort tests for push by using a _SmartClient.hooks['call'] hook.
316
class SmartClientHooks(hooks.Hooks):
317
5622.3.10 by Jelmer Vernooij
Don't require arguments to hooks.
318
    def __init__(self):
319
        hooks.Hooks.__init__(self, "bzrlib.smart.client", "_SmartClient.hooks")
5622.3.2 by Jelmer Vernooij
Add more lazily usable hook points.
320
        self.add_hook('call',
4119.3.2 by Robert Collins
Migrate existing hooks over to the new HookPoint infrastructure.
321
            "Called when the smart client is submitting a request to the "
322
            "smart server. Called with a bzrlib.smart.client.CallHookParams "
323
            "object. Streaming request bodies, and responses, are not "
5622.3.2 by Jelmer Vernooij
Add more lazily usable hook points.
324
            "accessible.", None)
325
326
5622.3.10 by Jelmer Vernooij
Don't require arguments to hooks.
327
_SmartClient.hooks = SmartClientHooks()
3703.3.3 by Andrew Bennetts
Neater effort tests for push by using a _SmartClient.hooks['call'] hook.
328
329
330
class CallHookParams(object):
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
331
3731.2.1 by Andrew Bennetts
Show total HPSS calls (if any) on stderr when -Dhpss is active.
332
    def __init__(self, method, args, body, readv_body, medium):
3703.3.3 by Andrew Bennetts
Neater effort tests for push by using a _SmartClient.hooks['call'] hook.
333
        self.method = method
334
        self.args = args
335
        self.body = body
336
        self.readv_body = readv_body
3731.2.1 by Andrew Bennetts
Show total HPSS calls (if any) on stderr when -Dhpss is active.
337
        self.medium = medium
3703.3.3 by Andrew Bennetts
Neater effort tests for push by using a _SmartClient.hooks['call'] hook.
338
339
    def __repr__(self):
340
        attrs = dict((k, v) for (k, v) in self.__dict__.iteritems()
341
                     if v is not None)
342
        return '<%s %r>' % (self.__class__.__name__, attrs)
343
344
    def __eq__(self, other):
345
        if type(other) is not type(self):
346
            return NotImplemented
347
        return self.__dict__ == other.__dict__
348
349
    def __ne__(self, other):
350
        return not self == other