For improving compression, it is possible to train a compression scheme with data which is known in advance to both the encoder and the decoder. If the new data to be sent is similar to the training data, you can get in some cases impressive compression gains.
You can see this effect "manually" by doing:
copy /b data+trainer appended
(or the equivalent UNIX command), then:
zipada -el3 trained.zip appended
on one hand, and:
zipada -el3 untrained.zip trainer data
on the other hand.
You'll see that "trained.zip" is smaller than "untrained.zip", although the contents are the same. If you need to send many "data" you don't yet know exactly, but that you know they will be similar to "trainer", you can leverage this information by informing once for all the encoder
and the decoder with the "trainer".
While this technique is available with some compression libraries, we provide a generic, standalone Trained_Compression package on which you can plug
any of your preferred compression schemes. An implementation is provided with our (equally generic & standalone) LZMA[.*] packages.
Check the "trained" directory
here or
here (revision 700+).
Here a schema of how it works:
Some results are shown here: