1
1
Bazaar Windows Shell Extension Options
2
======================================
2
========================================
4
4
.. contents:: :local:
9
This document details the implementation strategy chosen for the
9
This document details the imlpementation strategy chosen for the
10
10
Bazaar Windows Shell Extensions, otherwise known as TortoiseBzr, or TBZR.
11
11
As justification for the strategy, it also describes the general architecture
12
12
of Windows Shell Extensions, then looks at the C++ implemented TortoiseSvn
13
13
and the Python implemented TortoiseBzr, and discusses alternative
14
14
implementation strategies, and the reasons they were not chosen.
16
The following points summarize the strategy:
16
The following points summarize the strategy.
18
18
* Main shell extension code will be implemented in C++, and be as thin as
19
19
possible. It will not directly do any VCS work, but instead will perform
41
41
loaded by the Windows shell. There is no facility for shell extensions to
42
42
exist in a separate process - DLLs are the only option, and they are loaded
43
43
into other processes which take advantage of the Windows shell (although
44
obviously this DLL is free to do whatever it likes).
44
obviously this DLL is free to do whatever it likes)
46
46
For the sake of this discussion, there are 2 categories of shell extensions:
48
48
* Ones that create a new "namespace". The file-system itself is an example of
49
49
such a namespace, as is the "Recycle Bin". For a user-created example,
50
50
picture a new tree under "My Computer" which allows you to browse a remote
51
51
server - it creates a new, stand-alone tree that doesn't really interact
52
52
with the existing namespaces.
54
54
* Ones that enhance existing namespaces, including the filesystem. An example
55
55
would be an extension which uses Icon Overlays to modify how existing files
56
56
on disk are displayed or add items to their context menu, for example.
66
66
example, when notepad.exe first starts with an empty file it is using around
67
67
3.5MB of RAM. As soon as the FileOpen dialog is loaded, TortoiseSvn loads
68
68
well over 20 additional DLLs, including the MSVC8 runtime, into the Notepad
69
process causing its memory usage (as reported by task manager) to more than
70
double - all without doing anything tortoise specific at all. (In fairness,
71
this illustration is contrived - the code from these DLLs are already in
72
memory and there is no reason to suggest TSVN adds any other unreasonable
73
burden - but the general point remains valid.)
69
process causing its memory usage to more than double - all without doing
70
anything tortoise specific at all.
75
72
This has wide-ranging implications. It means that such shell extensions
76
73
should be developed using a tool which can never cause conflict with
77
74
arbitrary processes. For this very reason, MS recommend against using .NET
81
78
conflict badly with other Python implemented applications (and will certainly
82
79
kill them in some situations). A similar issue exists with GUI toolkits used
83
80
- using (say) PyGTK directly in the shell extension would need to be avoided
84
(which it currently is best I can tell). It should also be obvious that the
85
shell extension will be in many processes simultaneously, meaning use of a
86
simple log-file (for example) is problematic.
81
(which it currently is best I can tell). It should also be obvious the shell
82
extension will be in many processes simultaneously, meaning use of a simple
83
log-file etc is problematic.
88
85
In practice, there is only 1 truly safe option - a low-level language (such
89
86
as C/C++) which makes use of only the win32 API, and a static version of the
90
C runtime library if necessary. Obviously, this sucks from our POV. :)
87
C runtime library if necessary. Obviously, this sucks from our POV :)
92
89
[1]: http://blogs.msdn.com/oldnewthing/archive/2006/12/18/1317290.aspx
94
91
Analysis of TortoiseSVN code
95
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
92
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
97
94
TortoiseSVN is implemented in C++. It relies on an external process to
98
perform most UI (such as diff, log, commit etc.) commands, but it appears to
95
perform most UI (such as diff, log, commit etc commands), but it appears to
99
96
directly embed the SVN C libraries for the purposes of obtaining status for
100
97
icon overlays, context menu, drag&drop, etc.
102
99
The use of an external process to perform commands is fairly simplistic in
103
terms of parent and modal windows. For example, when selecting "Commit", a
100
terms of parent and modal windows - for example, when selecting "Commit", a
104
101
new process starts and *usually* ends up as the foreground window, but it may
105
102
occasionally be lost underneath the window which created it, and the user may
106
103
accidently start many processes when they only need 1. Best I can tell, this
111
108
directly needed by the shell are part of the "shell extension" and the rest
112
109
of TortoiseSvn is "just" a fairly large GUI application implementing many
113
110
commands. The command-line to the app has even been documented for people who
114
wish to automate tasks using that GUI. This GUI is also implemented in C++
115
using Windows resource files.
117
TortoiseSvn has an option (enabled by default) which enabled a cache using a
118
separate process, aptly named TSVNCache.exe. It uses a named pipe to accept
119
connections from other processes for various operations. When enabled, TSVN
120
fetches most (all?) status information from this process, but it also has the
121
option to talk directly to the VCS, along with options to disable functionality
111
wish to automate tasks using that GUI. This GUI appears to also be
112
implemented in C++ using Windows resource files.
114
TortoiseSvn appears to cache using a separate process, aptly named
115
TSVNCache.exe. It uses a named pipe to accept connections from other
116
processes for various operations. At this stage, it's still unclear exactly
117
what is fetched from the cache and exactly what the shell extension fetches
118
directly via the subversion C libraries.
124
120
There doesn't seem to be a good story for logging or debugging - which is
125
what you expect from C++ based apps. :( Most of the heavy lifting is done by
121
what you expect from C++ based apps :( Most of the heavy lifting is done by
126
122
the external application, which might offer better facilities.
128
124
Analysis of existing TortoiseBzr code
129
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
125
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
131
127
The existing code is actually quite cool given its history (SoC student,
132
128
etc), so this should not be taken as criticism of the implementer nor of the
133
implementation. Indeed, many criticisms are also true of the TortoiseSvn
129
implementation - indeed, many criticisms are also true of the TortoiseSvn
134
130
implementation - see above. However, I have attempted to list the bad things
135
131
rather than the good things so a clear future strategy can be agreed, with
136
132
all limitations understood.
162
158
and also error prone (it's possible the editor will check the file in,
163
159
meaning Windows explorer will be showing stale data). This may be possible to
164
160
address via file-system notifications, but a shared cache would be preferred
165
(although clearly more difficult to implement).
161
(although clearly more difficult to implement)
167
163
One tortoise port recently announced a technique for all tortoise ports to
168
164
share the same icon overlays to help work around a limitation in Windows on
169
the total number of overlays (it's limited to 15, due to the number of bits
165
the total number of overlays (its limited to 15, due to the number of bits
170
166
reserved in a 32bit int for overlays). TBZR needs to take advantage of that
171
167
(but to be fair, this overlay sharing technique was probably done after the
172
TBZR implementation).
174
170
The current code appears to recursively walk a tree to check if *any* file in
175
171
the tree has changed, so it can reflect this in the parent directory status.
176
172
This is almost certainly an evil thing to do (Shell Extensions are optimized
177
173
so that a folder doesn't even need to look in its direct children for another
178
174
folder, let alone recurse for any reason at all. It may be a network mounted
179
drive that doesn't perform at all.)
175
drive that doesn't perform at all)
181
177
Although somewhat dependent on bzr itself, we need a strategy for binary
182
178
releases (ie, it assumes python.exe, etc) and integration into an existing
302
298
external GUI apps themselves, etc) and see if a path forward does emerge for
303
299
Vista. We can re-evaluate this based on user feedback and more information
304
300
about features of the Vista property system.
309
The RPC mechanism and the tasks performed by the RPC server (rpc, file system
310
crawling and watching, device notifications, caching) are very similar to
311
those already implemented for TSVN and analysis of that code shows that
312
it is not particularly tied to any VCS model. As a result, consideration
313
should be given to making the best use of this existing debugged and
314
optimized technology.
316
Discussions with the TSVN developers have indicated that they would prefer us
317
to fork their code rather than introduce complexity and instability into
318
their code by attempting to share it. See the follow-ups to
319
http://thread.gmane.org/gmane.comp.version-control.subversion.tortoisesvn.devel/32635/focus=32651
322
For background, the TSVNCache process is fairly sophisticated - but mainly in
323
areas not related to source control. It has had various performance tweaks
324
and is smart in terms of minimizing its use of resources when possible. The
325
'cloc' utility counts ~5000 lines of C++ code and weighs in just under 200KB
326
on disk (not including headers), so this is not a trivial application.
327
However, the code that is of most interest (the crawlers, watchers and cache)
328
are roughly ~2500 lines of C++. Most of the source files only depend lightly
329
on SVN specifics, so it would not be a huge job to make the existing code
330
talk to Bazaar. The code is thread-safe, but not particularly thread-friendly
331
(ie, fairly coarse-grained locks are taken in most cases).
333
In practice, this give us 2 options - "fork" or "port":
335
* Fork the existing C++ code, replacing the existing source-control code with
336
code that talks to Bazaar. This would involve introducing a Python layer,
337
but only at the layers where we need to talk to bzrlib. The bulk of the
338
code would remain in C++.
340
This would have the following benefits:
342
- May offer significant performance advantages in some cases (eg, a
343
cache-hit would never enter Python at all.)
345
- Quickest time to a prototype working - the existing working code can be
348
And the following drawbacks:
350
- More complex to develop. People wishing to hack on it must be on Windows,
351
know C++ and own the most recent MSVC8.
353
- More complex to build and package: people making binaries must be on
354
Windows and have the most recent MSVC8.
356
- Is tied to Windows - it would be impractical for this to be
357
cross-platform, even just for test purposes (although parts of it
360
* Port the existing C++ code to Python. We would do this almost
361
"line-for-line", and attempt to keep many optimizations in place (or at
362
least document what the optimizations were for ones we consider dubious).
363
For the windows versions, pywin32 and ctypes would be leaned on - there
364
would be no C++ at all.
366
This would have the following benefits:
368
- Only need Python and Python skills to hack on it.
370
- No C++ compiler needed means easier to cut releases
372
- Python makes it easier to understand and maintain - it should appear much
373
less complex than the C++ version.
375
And the following drawbacks:
377
- Will be slower in some cases - eg, a cache-hit will involve executing
380
- Will take longer to get a minimal system working. In practice this
381
probably means the initial versions will not be as sophisticated.
383
Given the above, there are two issues which prevent Python being the clear
384
winner: (1) will it perform OK? (2) How much longer to a prototype?
386
My gut feeling on (1) is that it will perform fine, given a suitable Python
387
implementation. For example, Python code that simply looked up a dictionary
388
would be fast enough - so it all depends on how fast we can make our cache.
389
Re (2), it should be possible to have a "stub" process (did almost nothing in
390
terms of caching or crawling, but could be connected to by the shell) in a 8
391
hours, and some crawling and caching in 40. Note that this is separate from
392
the work included for the shell extension itself (the implementation of which
393
is largely independent of the TBZRCache implementation). So given the lack of
394
a deadline for any particular feature and the better long-term fit of using
395
Python, the conclusion is that we should "port" TSVN for bazaar.
397
Reuse of this code by Mercurial or other Python based VCS systems?
398
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
400
Incidentally, the hope is that this work can be picked up by the Mercurial
401
project (or anyone else who thinks it is of use). However, we will limit
402
ourselves to attempting to find a clean abstraction for the parts that talk
403
to the VCS (as good design would dictate regardless) and then try and assist
404
other projects in providing patches which work for both of us. In other
405
words, supporting multiple VCS systems is not an explicit goal at this stage,
406
but we would hope it is possible in the future.
411
305
The following is a high-level set of milestones for the implementation:
413
307
* Design the RPC mechanism used for icon overlays (ie, binary format used for
416
310
* Create Python prototype of the C++ "shim": modify the existing TBZR Python
417
311
code so that all references to "bzrlib" are removed. Implement the client
421
315
* Create initial implementation of RPC server in Python. This will use
422
316
bzrlib, but will also maintain a local cache to achieve the required
423
performance. File crawling and watching will not be implemented at this
424
stage, but caching will (although cache persistence might be skipped).
317
performance. The initial implementation may even be single-threaded, just
318
to keep synchronization issues to a minimum.
426
320
* Analyze performance of prototype. Verify that technique is feasible and
427
321
will offer reasonable performance and user experience.
429
* Implement file watching, crawling etc by "porting" TSVNCache code to
430
Python, as described above.
432
323
* Implement C++ shim: replace the Python prototype with a light-weight C++
433
version. We will fork the current TSVN sources, including its new
434
support for sharing icon overlays (although advice on how to setup this
324
version. We would work from the current TSVN sources, including its new
325
support for sharing icon overlays. Advice on if we should "fork" TSVN, or
326
try and manage our own svn based branch in bazaar are invited.
437
328
* Implement property pages and context menus in C++. Expand RPC server as
added
removed
doc/developers/tortoise-strategy.txt
