/brz/remove-bazaar : changes

» Changes from revision 3641.5.9

To get this branch, use:

bzr branch
http://gegoxaren.bato24.eu/bzr/brz/remove-bazaar

From Revision 3641.5.9 to 3641.3.21

expand all

Rev	Summary	Authors	Date
3641.5.9	Shrink the page size so that the three_level and iter_all Shrink the page size so that the three_level and iter_all tests don't have to use so many keys to get a three-level index.	John Arbash Meinel	17 years ago
3641.5.8	Fix up the test suite, now that we pack more efficiently. Fix up the test suite, now that we pack more efficiently. Unfortunately, that means upping the node count for the iter_all and three_level tests.	John Arbash Meinel	17 years ago
3641.5.7	Merge in bzr.dev to get the pyrex update Merge in bzr.dev to get the pyrex update	John Arbash Meinel	17 years ago
3641.5.6	Include the benchmarks for the new estimator. Include the benchmarks for the new estimator. All around, it seems to do a much better job. A bit faster, and a bit smaller.	John Arbash Meinel	17 years ago
3641.5.5	Document my attempt to use copy() as a look-ahead. Document my attempt to use copy() as a look-ahead.	John Arbash Meinel	17 years ago
3641.5.4	Using a different safety margin for the first repack, Using a different safety margin for the first repack, and using 2 repacks gives us effectively the same result, while still making it safe for arbitary data. (With 1-repack, it does effect the results 3-5%, and with 2-repacks the second margin gives the same results. Also, we now can get about 2-3:1 of lines that are 'blindly' added versus ones which are added with a SYNC.	John Arbash Meinel	17 years ago
3641.5.3	If we repack earlier, it catches this case. If we repack earlier, it catches this case. Still need to fix the other tests, but at least the too_much test passes now. Impact on real-world results is measurable (2-3% final compression). Is it worth it?	John Arbash Meinel	17 years ago
3641.5.2	(broken, but hopeful) Change the compact logic. (broken, but hopeful) Change the compact logic. Instead of only paying attention to the total bytes read, use the fact that we know some of that is already compressed well. So instead, we just pay attention to the bytes that are added since the last sync. This means we Z_SYNC_FLUSH much less often. (After syncing, we end up with more room to add without syncing.) This improves both the time to compress and the final compressed size. Need to update tests with the new offsets. Also, we seem to have found a case where using Z_SYNC_FLUSH in the middle of a stream will actually generate better compression than compressing the whole stream in one pass. test_too_much_data_does_not_exceed_size triggers this. It can be packed in more than 100 bytes less than the amount given by a full compress.	John Arbash Meinel	17 years ago
3641.5.1	Update the stats for the current code layout. Update the stats for the current code layout. This shows why I like _max_repack=2 so much. It is the highest value that has 'no waste'. At _max_repack=2, you can always sneak in 1 more line, which avoids triggering an extra repack. Also, updating the timings with the current tuning.	John Arbash Meinel	17 years ago
3641.3.31	NEWS NEWS	John Arbash Meinel	17 years ago
3641.3.30	Rename _parse_btree to _btree_serializer Rename _parse_btree to _btree_serializer	John Arbash Meinel	17 years ago
3641.3.29	Cleanup the copyright headers Cleanup the copyright headers	John Arbash Meinel	17 years ago
3641.3.28	Remove FixedMemoryGraphIndex Remove FixedMemoryGraphIndex	John Arbash Meinel	17 years ago
3641.3.27	Bringing reserved in as a keyword to write() also saves some... Bringing reserved in as a keyword to write() also saves some time. ChunkWriter.write() is called for every key, so having an extra redirection The was a measurable loss. Also, a small tuneup for the compressor that we access often. Unfortunately, there isn't much more tuning that we can do in that function.	John Arbash Meinel	17 years ago
3641.3.26	A couple small tweaks. A couple small tweaks.	John Arbash Meinel	17 years ago
3641.3.25	Shave off some more time by using exact accessors. Shave off some more time by using exact accessors. Basically, the inner access is using PyTuple and PyString functions instead of their PyObject counterparts.	John Arbash Meinel	17 years ago
3641.3.24	Use the compiled flatten function. Use the compiled flatten function. Shows a 3-5% gain in testing and real-world conditions.	John Arbash Meinel	17 years ago
3641.3.23	Flattened all the way down the stack. Flattened all the way down the stack. Next might be to try using something other than plain iteration. Like using PySequenceFast or some variant thereof.	John Arbash Meinel	17 years ago
3641.3.22	flatten the next level flatten the next level	John Arbash Meinel	17 years ago
3641.3.21	Flatten the outermost str.join() into memcpy's Flatten the outermost str.join() into memcpy's	John Arbash Meinel	17 years ago

Older »