bpe_compressor is a simple compressor for .txt files using BPE algorithm.
It's not meant to be used in production.
.\build.bat$ .\main.exe zip input.txt output.bpeor
$ .\main.exe unzip input.bpe output.txt_______________________________________
|BPE(version)(LE|BE) |
| |
|(compressed data length as uint32_t) |
|____________________________ |
| | |
| | |
| (compressed data | |
| as linked list) | |
| | |
|____________________________| |
| (highest element value in the |
| compressed data as uint32_t) |
| |
| (pairs array length as uint32_t) |
| |
|____________________________ |
| | |
| | |
| (pairs array | |
| as uint32_t[]) | |
| | |
|____________________________| |
| |
| EOF |
|--------------------------------------|Byte-pair encoding (Wikipedia)