mardi 26 avril 2016

BNS - monnaie centrale - état de mars 2016

En graphiques, sans blabla !

dimanche 3 avril 2016

Zip-Ada v.50

There is a new version of Zip-Ada @ .


In a nutshell, there are now, finally, fast *and* efficient compression methods available.

* Changes in '50', 31-Mar-2016:
  - Zip.Compress.Shrink is slightly faster
  - Zip.Compress.Deflate has new compression features:
     - Deflate_Fixed is much faster, with slightly better compression
     - Deflate_1 was added: strength similar to zlib, level 6
     - Deflate_2 was added: strength similar to zlib, level 9
     - Deflate_3 was added: strength similar to 7-Zip, method=deflate, level 5

I use the term "similar" because it the compression strength depends on the algorithms used and on the data, so it may differ from case to case. In the following charts, we have a comparison on the two most known benchmark data set ("corpora"), where the similarity with zlib (=info-zip, prefix iz_ below) holds, but not at all with 7-Zip-with-Deflate.
In blue, you see non-Deflate formats (BZip2 and LZMA), just to remind that the world doesn't stop with Deflate, although it's the topic in this article.
In green, you have Zip archives made by Zip-Ada.

Click to enlarge image
Click to enlarge image

Here is the biggest surprise I've had by testing randomly chosen data: a 162MB sparse integer matrix (among a bunch of results for a Kaggle challenge) which is a very redundant data. First, 7-Zip in Deflate mode gives a comparatively poor compression ratio - don't worry for 7-Zip, the LZMA mode, genuine to 7-Zip, is second best in the list. The most surprising aspect is that the Shrink format (LZW algorithm) has a compressed size only 5.6% larger than the best Deflate (here, KZip).

Click to enlarge image

Typically the penalty for LZW (used for GIF images) is from 25% to 100% compared to the best Deflate (used for PNG images). Of course, at the other end of redundancy spectrum, data which are closer to random are also more difficult to compress and the differences between LZW and Deflate narrow forcefully.

About Deflate

As you perhaps know, the Deflate format, invented around 1989 by the late Phil Katz for his PKZip program, performs compression in two steps by combining a LZ77 algorithm with Huffman encoding.
In this edition of Zip-Ada, two known algorithms (one for LZ77, one for finding an appropriate Huffman encoding based on an alphabet's statistics) are combined probably for the first time within the same software.
Additionally, the determination of compressed blocks' boundaries is done by an original algorithm (the Taillaule algorithm) based on similarities between Huffman code sets.


Zip-Ada is a library for dealing with the Zip compressed archive
file format. It supplies:

 - compression with the following sub-formats ("methods"):
     Store, Reduce, Shrink (LZW) and Deflate
 - decompression for the following sub-formats ("methods"):
     Store, Reduce, Shrink (LZW), Implode, Deflate, BZip2 and LZMA
 - encryption and decryption (portable Zip 2.0 encryption scheme)
 - unconditional portability - within limits of compiler's provided
     integer types and target architecture capacity
 - input (archive to decompress or data to compress) can be any data stream
 - output (archive to build or data to extract) can be any data stream
 - types Zip_info and Zip_Create_info to handle archives quickly and easily
 - cross format compatibility with the most various tools and file formats
     based on the Zip format: 7-zip, Info-Zip's Zip, WinZip, PKZip, Java's
     JARs, OpenDocument files, MS Office 2007+, Nokia themes, and many others
 - task safety: this library can be used ad libitum in parallel processing
 - endian-neutral I/O


mardi 23 février 2016

Hoarding cash

Just came across this Zerohedge article: "Safes Sell Out In Japan, 1,000 Franc Note Demand Soars As NIRP Triggers Cash Hoarding".

Hoarding cash may sound a like a good idea currently, with low price inflation and negative interest rates.
However, it is a big mistake to put Swiss Francs as cash into a safe. You may forget about it and have a bad surprise when you want to use or convert your nice pieces of paper in the future.
Here is the explanation in four charts.

1) Base money and 1,000 CHF banknotes (y-scale limited to 80Bn - click to enlarge chart)

The amount of cash hoarded as 1,000 CHF notes (CHF 71 Billion) is now much larger than the entire monetary base as in summer 2008 (CHF 45 Billion). First "Uh-oh". In case you are thinking "So, what ?", ask yourself what might be the value of this additional cash. The answer is: nothing. Again: this cash is backed by nothing. The market and especially the people hoarding cash into safes are not (yet) aware of that, fortunately for them (for now).
Now, what is the entire chart of base money (in the chart above, the y-scale is limited to 80 Billion) ?

2) CHF Base money (log scale - click to enlarge chart)
Second "Uh-oh". Again, this new money is backed by nothing. It is sold against foreign currencies by the Swiss National Bank to Mr Market who is thinking "Wow, the Swiss Franc, it must be rock-solid! The mountains, the gold, the army - what can be wrong ?"
With the foreign currencies, the SNB is buying bonds and equities in EUR or USD with more or less success.
Oh, a detail. The y-scale of chart 2 is logarithmic: there is a doubling at the crossing of each horizontal line.
With a normal scale, it looks like this:
3) CHF Base money (lin scale - click to enlarge chart)
Now a last one, if you did not have enough charts:

4) CHF and ARS Base money (pre mid-2014: one data point per year for ARS - click to enlarge chart)
Happy hoarding!

vendredi 19 février 2016

Visual Deflating

Compressed as bitmap with the Deflate format with dynamically computed Huffman trees, you may get this when dumping the bit length for each code (columns) with successive chunks of the image (rows):

Literals / LZ Lengths / LZ Distances
Current research with the Zip-Ada project.

mercredi 17 février 2016


...not in the economical sense fortunately, except that it's all about economy of bits.

Following last post about limited-length Huffman trees, there is now the very early development of a compression algorithm for the so-called "Dynamic Deflate" compression format. From revision #297 the code (checkout here) seems to correctly compress data (correctly means that the data is correctly decoded, even by buggy but popular decoders...).
The efficiciency of the compression is in the works. Stay tuned!

Testing is welcome: build and use the ZipAda command-line tool, or use the Deflate_1 method in your code using the Zip-Ada library.

mercredi 10 février 2016

Playing with limited-length Huffman trees

A small portion of a Huffman tree
Proper counting is crucial

On the second image you can recognize "DIZAINES" and "UNITÉS". The letters are hardly visible after some 90 years of use... It's a pool table in the mythic Schlauch Restaurant in Zurich.

Back to Huffman trees: just came across a beautiful algorithm [1] for building a Huffman tree with, as a constraint, a limited code length. With the original Huffman algorithm, parts of the tree may become arbitrarily long - well, obviously not longer that the alphabet to be encoded. This algorithm is optimal (given the constraint), fast and very economical in terms of used memory.

There is an equally beautiful implementation [2], translated for the purpose of the Zip-Ada project - a proper dynamic Deflate for compression is still missing, and a limited-length Huffman tree building algorithm is needed for that.
Translation is there: specification, body, test procedure, and abstract enough to build with an Ada 83 compiler (at least GNAT in -gnat83 mode)!
[1] Search: "A Fast and Space-Economical Algorithm for Length-Limited Coding"
[2] Search: "katajainen.c" (this part of the "Zopfli" project)

Avindex 1.0, snapshot #8, "Rock-solid balance sheet" edition

(Click to enlarge chart)

We learn today that Deutsche Bank's balance sheet is "rock-solid".
So, don't worry, folks!

One upon a time there was a reinsurance company called Converium. The CEO said exactly that: "Converium has a rock-solid balance sheet". It became a quickly running gag among the firm's employees...