init
This commit is contained in:
13
expkg/vendor/lz4/examples/.gitignore
vendored
Normal file
13
expkg/vendor/lz4/examples/.gitignore
vendored
Normal file
@ -0,0 +1,13 @@
|
||||
/Makefile.lz4*
|
||||
/print_version
|
||||
/simple_buffer
|
||||
/frameCompress
|
||||
/fileCompress
|
||||
/blockStreaming_doubleBuffer
|
||||
/blockStreaming_ringBuffer
|
||||
/streamingHC_ringBuffer
|
||||
/blockStreaming_lineByLine
|
||||
/dictionaryRandomAccess
|
||||
/bench_functions
|
||||
/*.exe
|
||||
/*.lz4s*
|
||||
83
expkg/vendor/lz4/examples/COPYING
vendored
Normal file
83
expkg/vendor/lz4/examples/COPYING
vendored
Normal file
@ -0,0 +1,83 @@
|
||||
Full name:
|
||||
GNU General Public License v2.0 or later
|
||||
|
||||
Short identifier:
|
||||
GPL-2.0-or-later
|
||||
|
||||
Text:
|
||||
GNU GENERAL PUBLIC LICENSE
|
||||
Version 2, June 1991
|
||||
|
||||
Copyright (C) 1989, 1991 Free Software Foundation, Inc.
|
||||
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA
|
||||
|
||||
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things.
|
||||
|
||||
To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it.
|
||||
|
||||
For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.
|
||||
|
||||
We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software.
|
||||
|
||||
Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations.
|
||||
|
||||
Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all.
|
||||
|
||||
The precise terms and conditions for copying, distribution and modification follow.
|
||||
|
||||
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
|
||||
|
||||
0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you".
|
||||
Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does.
|
||||
|
||||
1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program.
|
||||
You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee.
|
||||
|
||||
2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions:
|
||||
a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change.
|
||||
b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License.
|
||||
c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.)
|
||||
These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it.
|
||||
|
||||
Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program.
|
||||
|
||||
In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License.
|
||||
|
||||
3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following:
|
||||
a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or,
|
||||
b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or,
|
||||
c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.)
|
||||
The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable.
|
||||
|
||||
If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code.
|
||||
|
||||
4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance.
|
||||
|
||||
5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it.
|
||||
|
||||
6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License.
|
||||
|
||||
7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program.
|
||||
If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances.
|
||||
|
||||
It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice.
|
||||
|
||||
This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License.
|
||||
|
||||
8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License.
|
||||
|
||||
9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns.
|
||||
Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation.
|
||||
|
||||
10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally.
|
||||
NO WARRANTY
|
||||
|
||||
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
|
||||
103
expkg/vendor/lz4/examples/Makefile
vendored
Normal file
103
expkg/vendor/lz4/examples/Makefile
vendored
Normal file
@ -0,0 +1,103 @@
|
||||
# ##########################################################################
|
||||
# LZ4 examples - Makefile
|
||||
# Copyright (C) Yann Collet 2011-2020
|
||||
#
|
||||
# GPL v2 License
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License along
|
||||
# with this program; if not, write to the Free Software Foundation, Inc.,
|
||||
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
|
||||
#
|
||||
# You can contact the author at :
|
||||
# - LZ4 source repository : https://github.com/lz4/lz4
|
||||
# - LZ4 forum froup : https://groups.google.com/forum/#!forum/lz4c
|
||||
# ##########################################################################
|
||||
# This makefile compile and test
|
||||
# example programs, using (mostly) LZ4 streaming library,
|
||||
# kindly provided by Takayuki Matsuoka
|
||||
# ##########################################################################
|
||||
|
||||
LIBDIR := ../lib
|
||||
|
||||
CPPFLAGS += -I$(LIBDIR)
|
||||
USERCFLAGS:= $(CFLAGS)
|
||||
WFLAGS = -std=gnu99 -Wall -Wextra -Wundef -Wshadow -Wcast-align -Wstrict-prototypes -Wc++-compat
|
||||
CFLAGS = $(WFLAGS) -O2 $(USERCFLAGS)
|
||||
|
||||
TESTFILE = Makefile
|
||||
TESTFILE_SMALL= .gitignore
|
||||
LZ4DIR = ../programs
|
||||
LZ4 = $(LZ4DIR)/lz4
|
||||
LIBLZ4DIR = ../lib
|
||||
LIBLZ4SRCS = $(notdir $(wildcard $(LIBLZ4DIR)/*.c))
|
||||
LIBLZ4OBJS = $(LIBLZ4SRCS:.c=.o)
|
||||
SLIBLZ4 := liblz4.a
|
||||
|
||||
default: all
|
||||
|
||||
C_SRCDIRS = . ../lib
|
||||
include ../build/make/multiconf.make
|
||||
|
||||
$(SLIBLZ4):
|
||||
$(eval $(call static_library,$(SLIBLZ4), $(LIBLZ4OBJS)))
|
||||
|
||||
ALL = print_version \
|
||||
simple_buffer \
|
||||
frameCompress \
|
||||
fileCompress \
|
||||
blockStreaming_doubleBuffer \
|
||||
blockStreaming_ringBuffer \
|
||||
streamingHC_ringBuffer \
|
||||
blockStreaming_lineByLine \
|
||||
dictionaryRandomAccess \
|
||||
bench_functions
|
||||
|
||||
.PHONY: all
|
||||
all: $(ALL)
|
||||
|
||||
$(ALL): $(SLIBLZ4)
|
||||
|
||||
.PHONY:$(LZ4)
|
||||
$(LZ4) :
|
||||
$(MAKE) -j -C $(LZ4DIR) lz4
|
||||
|
||||
.PHONY: test
|
||||
test : all $(LZ4)
|
||||
@echo "\n=== Print Version ==="
|
||||
./print_version$(EXT)
|
||||
@echo "\n=== Simple compression example ==="
|
||||
./simple_buffer$(EXT)
|
||||
@echo "\n=== Double-buffer ==="
|
||||
./blockStreaming_doubleBuffer$(EXT) $(TESTFILE)
|
||||
@echo "\n=== Ring Buffer ==="
|
||||
./blockStreaming_ringBuffer$(EXT) $(TESTFILE)
|
||||
@echo "\n=== Ring Buffer + LZ4 HC ==="
|
||||
./streamingHC_ringBuffer$(EXT) $(TESTFILE)
|
||||
@echo "\n=== Compress line by line ==="
|
||||
./blockStreaming_lineByLine$(EXT) $(TESTFILE)
|
||||
@echo "\n=== Dictionary Random Access ==="
|
||||
./dictionaryRandomAccess$(EXT) $(TESTFILE) $(TESTFILE) 1100 1400
|
||||
./dictionaryRandomAccess$(EXT) $(TESTFILE_SMALL) $(TESTFILE_SMALL) 0 32
|
||||
@echo "\n=== Frame compression ==="
|
||||
./frameCompress$(EXT) $(TESTFILE)
|
||||
$(LZ4) -vt $(TESTFILE).lz4
|
||||
@echo "\n=== file compression ==="
|
||||
./fileCompress$(EXT) $(TESTFILE)
|
||||
$(LZ4) -vt $(TESTFILE).lz4
|
||||
@echo "\n=== Q&D benchmark ==="
|
||||
./bench_functions$(EXT) 10000
|
||||
|
||||
.PHONY: clean
|
||||
clean:
|
||||
@$(RM) core *.o *.dec *-0 *-9 *-8192 *.lz4s *.lz4 $(ALL)
|
||||
@echo Cleaning completed
|
||||
11
expkg/vendor/lz4/examples/README.md
vendored
Normal file
11
expkg/vendor/lz4/examples/README.md
vendored
Normal file
@ -0,0 +1,11 @@
|
||||
# LZ4 examples
|
||||
|
||||
All examples are GPL-v2 licensed.
|
||||
|
||||
## Documents
|
||||
|
||||
- [Streaming API Basics](streaming_api_basics.md)
|
||||
- Examples
|
||||
- [Double Buffer](blockStreaming_doubleBuffer.md)
|
||||
- [Line by Line Text Compression](blockStreaming_lineByLine.md)
|
||||
- [Dictionary Random Access](dictionaryRandomAccess.md)
|
||||
373
expkg/vendor/lz4/examples/bench_functions.c
vendored
Normal file
373
expkg/vendor/lz4/examples/bench_functions.c
vendored
Normal file
@ -0,0 +1,373 @@
|
||||
/*
|
||||
* bench_functions.c
|
||||
* Copyright : Kyle Harper
|
||||
* License : Follows same licensing as the lz4.c/lz4.h program at any given time. Currently, BSD 2.
|
||||
* Description: A program to demonstrate the various compression functions involved in when using LZ4_compress_default(). The idea
|
||||
* is to show how each step in the call stack can be used directly, if desired. There is also some benchmarking for
|
||||
* each function to demonstrate the (probably lack of) performance difference when jumping the stack.
|
||||
* (If you're new to lz4, please read simple_buffer.c to understand the fundamentals)
|
||||
*
|
||||
* The call stack (before theoretical compiler optimizations) for LZ4_compress_default is as follows:
|
||||
* LZ4_compress_default
|
||||
* LZ4_compress_fast
|
||||
* LZ4_compress_fast_extState
|
||||
* LZ4_compress_generic
|
||||
*
|
||||
* LZ4_compress_default()
|
||||
* This is the recommended function for compressing data. It will serve as the baseline for comparison.
|
||||
* LZ4_compress_fast()
|
||||
* Despite its name, it's not a "fast" version of compression. It simply decides if HEAPMODE is set and either
|
||||
* allocates memory on the heap for a struct or creates the struct directly on the stack. Stack access is generally
|
||||
* faster but this function itself isn't giving that advantage, it's just some logic for compile time.
|
||||
* LZ4_compress_fast_extState()
|
||||
* This simply accepts all the pointers and values collected thus far and adds logic to determine how
|
||||
* LZ4_compress_generic should be invoked; specifically: can the source fit into a single pass as determined by
|
||||
* LZ4_64Klimit.
|
||||
* LZ4_compress_generic()
|
||||
* As the name suggests, this is the generic function that ultimately does most of the heavy lifting. Calling this
|
||||
* directly can help avoid some test cases and branching which might be useful in some implementation-specific
|
||||
* situations, but you really need to know what you're doing AND what you're asking lz4 to do! You also need a
|
||||
* wrapper function because this function isn't exposed with lz4.h.
|
||||
*
|
||||
* The call stack for decompression functions is shallow. There are 2 options:
|
||||
* LZ4_decompress_safe || LZ4_decompress_fast
|
||||
* LZ4_decompress_generic
|
||||
*
|
||||
* LZ4_decompress_safe
|
||||
* This is the recommended function for decompressing data. It is considered safe because the caller specifies
|
||||
* both the size of the compressed buffer to read as well as the maximum size of the output (decompressed) buffer
|
||||
* instead of just the latter.
|
||||
* LZ4_decompress_fast
|
||||
* Again, despite its name it's not a "fast" version of decompression. It simply frees the caller of sending the
|
||||
* size of the compressed buffer (it will simply be read-to-end, hence it's non-safety).
|
||||
* LZ4_decompress_generic
|
||||
* This is the generic function that both of the LZ4_decompress_* functions above end up calling. Calling this
|
||||
* directly is not advised, period. Furthermore, it is a static inline function in lz4.c, so there isn't a symbol
|
||||
* exposed for anyone using lz4.h to utilize.
|
||||
*
|
||||
* Special Note About Decompression:
|
||||
* Using the LZ4_decompress_safe() function protects against malicious (user) input. If you are using data from a
|
||||
* trusted source, or if your program is the producer (P) as well as its consumer (C) in a PC or MPMC setup, you can
|
||||
* safely use the LZ4_decompress_fast function.
|
||||
*/
|
||||
|
||||
/* Since lz4 compiles with c99 and not gnu/std99 we need to enable POSIX linking for time.h structs and functions. */
|
||||
#if __STDC_VERSION__ >= 199901L
|
||||
#define _XOPEN_SOURCE 600
|
||||
#else
|
||||
#define _XOPEN_SOURCE 500
|
||||
#endif
|
||||
#define _POSIX_C_SOURCE 199309L
|
||||
|
||||
/* Includes, for Power! */
|
||||
#define LZ4_DISABLE_DEPRECATE_WARNINGS /* LZ4_decompress_fast */
|
||||
#include "lz4.h"
|
||||
#include <stdio.h> /* for printf() */
|
||||
#include <stdlib.h> /* for exit() */
|
||||
#include <string.h> /* for atoi() memcmp() */
|
||||
#include <stdint.h> /* for uint_types */
|
||||
#include <inttypes.h> /* for PRIu64 */
|
||||
#include <time.h> /* for clock() */
|
||||
#include <locale.h> /* for setlocale() */
|
||||
#include <limits.h> /* for INT_MAX */
|
||||
#include <assert.h>
|
||||
|
||||
/* We need to know what one billion is for clock timing. */
|
||||
#define BILLION 1000000000L
|
||||
|
||||
/* Create a crude set of test IDs so we can switch on them later (Can't switch() on a char[] or char*). */
|
||||
#define ID__LZ4_COMPRESS_DEFAULT 1
|
||||
#define ID__LZ4_COMPRESS_FAST 2
|
||||
#define ID__LZ4_COMPRESS_FAST_EXTSTATE 3
|
||||
#define ID__LZ4_COMPRESS_GENERIC 4
|
||||
#define ID__LZ4_DECOMPRESS_SAFE 5
|
||||
#define ID__LZ4_DECOMPRESS_FAST 6
|
||||
|
||||
|
||||
/*
|
||||
* Easy show-error-and-bail function.
|
||||
*/
|
||||
void run_screaming(const char *message, const int code) {
|
||||
printf("%s \n", message);
|
||||
exit(code);
|
||||
}
|
||||
|
||||
/*
|
||||
* Centralize the usage function to keep main cleaner.
|
||||
*/
|
||||
void usage(const char* exeName, const char* message) {
|
||||
printf("Usage: %s <iterations> \n", exeName);
|
||||
run_screaming(message, 1);
|
||||
return;
|
||||
}
|
||||
|
||||
#define CHECK(c) { if (!(c)) { run_screaming(#c, 1); } }
|
||||
|
||||
|
||||
/*
|
||||
* Runs the benchmark for LZ4_* function based on function_id.
|
||||
* @return : benchmark duration, in ns
|
||||
*/
|
||||
uint64_t bench(
|
||||
const char *known_good_dst,
|
||||
const int function_id,
|
||||
int iterations,
|
||||
const char *src,
|
||||
char *dst,
|
||||
const size_t src_size,
|
||||
const size_t max_dst_size,
|
||||
const size_t comp_size
|
||||
) {
|
||||
int rv = 0;
|
||||
const int warm_up = 5000;
|
||||
const int acceleration = 1;
|
||||
LZ4_stream_t state;
|
||||
clock_t start = clock();
|
||||
|
||||
// Select the right function to perform the benchmark on. We perform 5000 initial loops to warm the cache and ensure that dst
|
||||
// remains matching to known_good_dst between successive calls.
|
||||
switch(function_id) {
|
||||
case ID__LZ4_COMPRESS_DEFAULT:
|
||||
printf("Starting benchmark for function: LZ4_compress_default()\n");
|
||||
for(int junk=0; junk<warm_up; junk++)
|
||||
rv = LZ4_compress_default(src, dst, (int)src_size, (int)max_dst_size);
|
||||
if (rv < 1)
|
||||
run_screaming("Couldn't run LZ4_compress_default()... error code received is in exit code.", rv);
|
||||
if (memcmp(known_good_dst, dst, max_dst_size) != 0)
|
||||
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
|
||||
start = clock();
|
||||
for (int i=1; i<=iterations; i++)
|
||||
LZ4_compress_default(src, dst, (int)src_size, (int)max_dst_size);
|
||||
break;
|
||||
|
||||
case ID__LZ4_COMPRESS_FAST:
|
||||
printf("Starting benchmark for function: LZ4_compress_fast()\n");
|
||||
for(int junk=0; junk<warm_up; junk++)
|
||||
rv = LZ4_compress_fast(src, dst, (int)src_size, (int)max_dst_size, acceleration);
|
||||
if (rv < 1)
|
||||
run_screaming("Couldn't run LZ4_compress_fast()... error code received is in exit code.", rv);
|
||||
if (memcmp(known_good_dst, dst, max_dst_size) != 0)
|
||||
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
|
||||
start = clock();
|
||||
for (int i=1; i<=iterations; i++)
|
||||
LZ4_compress_fast(src, dst, (int)src_size, (int)max_dst_size, acceleration);
|
||||
break;
|
||||
|
||||
case ID__LZ4_COMPRESS_FAST_EXTSTATE:
|
||||
printf("Starting benchmark for function: LZ4_compress_fast_extState()\n");
|
||||
for(int junk=0; junk<warm_up; junk++)
|
||||
rv = LZ4_compress_fast_extState(&state, src, dst, (int)src_size, (int)max_dst_size, acceleration);
|
||||
if (rv < 1)
|
||||
run_screaming("Couldn't run LZ4_compress_fast_extState()... error code received is in exit code.", rv);
|
||||
if (memcmp(known_good_dst, dst, max_dst_size) != 0)
|
||||
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
|
||||
start = clock();
|
||||
for (int i=1; i<=iterations; i++)
|
||||
LZ4_compress_fast_extState(&state, src, dst, (int)src_size, (int)max_dst_size, acceleration);
|
||||
break;
|
||||
|
||||
// Disabled until LZ4_compress_generic() is exposed in the header.
|
||||
// case ID__LZ4_COMPRESS_GENERIC:
|
||||
// printf("Starting benchmark for function: LZ4_compress_generic()\n");
|
||||
// LZ4_resetStream((LZ4_stream_t*)&state);
|
||||
// for(int junk=0; junk<warm_up; junk++) {
|
||||
// LZ4_resetStream((LZ4_stream_t*)&state);
|
||||
// //rv = LZ4_compress_generic_wrapper(&state, src, dst, src_size, max_dst_size, notLimited, byU16, noDict, noDictIssue, acceleration);
|
||||
// LZ4_compress_generic_wrapper(&state, src, dst, src_size, max_dst_size, acceleration);
|
||||
// }
|
||||
// if (rv < 1)
|
||||
// run_screaming("Couldn't run LZ4_compress_generic()... error code received is in exit code.", rv);
|
||||
// if (memcmp(known_good_dst, dst, max_dst_size) != 0)
|
||||
// run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
|
||||
// for (int i=1; i<=iterations; i++) {
|
||||
// LZ4_resetStream((LZ4_stream_t*)&state);
|
||||
// //LZ4_compress_generic_wrapper(&state, src, dst, src_size, max_dst_size, notLimited, byU16, noDict, noDictIssue, acceleration);
|
||||
// LZ4_compress_generic_wrapper(&state, src, dst, src_size, max_dst_size, acceleration);
|
||||
// }
|
||||
// break;
|
||||
|
||||
case ID__LZ4_DECOMPRESS_SAFE:
|
||||
printf("Starting benchmark for function: LZ4_decompress_safe()\n");
|
||||
for(int junk=0; junk<warm_up; junk++)
|
||||
rv = LZ4_decompress_safe(src, dst, (int)comp_size, (int)src_size);
|
||||
if (rv < 1)
|
||||
run_screaming("Couldn't run LZ4_decompress_safe()... error code received is in exit code.", rv);
|
||||
if (memcmp(known_good_dst, dst, src_size) != 0)
|
||||
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
|
||||
start = clock();
|
||||
for (int i=1; i<=iterations; i++)
|
||||
LZ4_decompress_safe(src, dst, (int)comp_size, (int)src_size);
|
||||
break;
|
||||
|
||||
case ID__LZ4_DECOMPRESS_FAST:
|
||||
printf("Starting benchmark for function: LZ4_decompress_fast()\n");
|
||||
for(int junk=0; junk<warm_up; junk++)
|
||||
rv = LZ4_decompress_fast(src, dst, (int)src_size);
|
||||
if (rv < 1)
|
||||
run_screaming("Couldn't run LZ4_decompress_fast()... error code received is in exit code.", rv);
|
||||
if (memcmp(known_good_dst, dst, src_size) != 0)
|
||||
run_screaming("According to memcmp(), the compressed dst we got doesn't match the known_good_dst... ruh roh.", 1);
|
||||
start = clock();
|
||||
for (int i=1; i<=iterations; i++)
|
||||
LZ4_decompress_fast(src, dst, (int)src_size);
|
||||
break;
|
||||
|
||||
default:
|
||||
run_screaming("The test specified isn't valid. Please check your code.", 1);
|
||||
break;
|
||||
}
|
||||
|
||||
{ clock_t end = clock();
|
||||
|
||||
// Low resolution timer => requires more iterations to measure something
|
||||
if (end == start) {
|
||||
assert(iterations < (INT_MAX / 10));
|
||||
iterations *= 10;
|
||||
printf("not enough iterations => increase nb of iterations to %i \n", iterations);
|
||||
return bench(known_good_dst, function_id, iterations, src, dst, src_size, max_dst_size, comp_size);
|
||||
}
|
||||
|
||||
return (uint64_t)((double)(end - start) / CLOCKS_PER_SEC * BILLION);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* main()
|
||||
* We will demonstrate the use of each function for simplicity sake. Then we will run 2 suites of benchmarking:
|
||||
* Test suite A) Uses generic Lorem Ipsum text which should be generally compressible insomuch as basic human text is
|
||||
* compressible for such a small src_size
|
||||
* Test Suite B) For the sake of testing, see what results we get if the data is drastically easier to compress. IF there are
|
||||
* indeed losses and IF more compressible data is faster to process, this will exacerbate the findings.
|
||||
*/
|
||||
int main(int argc, char **argv) {
|
||||
// Get and verify options. There's really only 1: How many iterations to run.
|
||||
const char* exeName = argv[0];
|
||||
int iterations = 1000000;
|
||||
if (argc > 1)
|
||||
iterations = atoi(argv[1]);
|
||||
if (iterations < 1)
|
||||
usage(exeName, "Argument 1 (iterations) must be > 0.");
|
||||
|
||||
// First we will create 2 sources (char *) of 2000 bytes each. One normal text, the other highly-compressible text.
|
||||
const char src[] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed luctus purus et risus vulputate, et mollis orci ullamcorper. Nulla facilisi. Fusce in ligula sed purus varius aliquet interdum vitae justo. Proin quis diam velit. Nulla varius iaculis auctor. Cras volutpat, justo eu dictum pulvinar, elit sem porttitor metus, et imperdiet metus sapien et ante. Nullam nisi nulla, ornare eu tristique eu, dignissim vitae diam. Nulla sagittis porta libero, a accumsan felis sagittis scelerisque. Integer laoreet eleifend congue. Etiam rhoncus leo vel dolor fermentum, quis luctus nisl iaculis. Praesent a erat sapien. Aliquam semper mi in lorem ultrices ultricies. Lorem ipsum dolor sit amet, consectetur adipiscing elit. In feugiat risus sed enim ultrices, at sodales nulla tristique. Maecenas eget pellentesque justo, sed pellentesque lectus. Fusce sagittis sit amet elit vel varius. Donec sed ligula nec ligula vulputate rutrum sed ut lectus. Etiam congue pharetra leo vitae cursus. Morbi enim ante, porttitor ut varius vel, tincidunt quis justo. Nunc iaculis, risus id ultrices semper, metus est efficitur ligula, vel posuere risus nunc eget purus. Ut lorem turpis, condimentum at sem sed, porta aliquam turpis. In ut sapien a nulla dictum tincidunt quis sit amet lorem. Fusce at est egestas, luctus neque eu, consectetur tortor. Phasellus eleifend ultricies nulla ac lobortis. Morbi maximus quam cursus vehicula iaculis. Maecenas cursus vel justo ut rutrum. Curabitur magna orci, dignissim eget dapibus vitae, finibus id lacus. Praesent rhoncus mattis augue vitae bibendum. Praesent porta mauris non ultrices fermentum. Quisque vulputate ipsum in sodales pulvinar. Aliquam nec mollis felis. Donec vitae augue pulvinar, congue nisl sed, pretium purus. Fusce lobortis mi ac neque scelerisque semper. Pellentesque vel est vitae magna aliquet aliquet. Nam non dolor. Nulla facilisi. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Morbi ac lacinia felis metus.";
|
||||
const char hc_src[] = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
|
||||
// Set and derive sizes. Since we're using strings, use strlen() + 1 for \0.
|
||||
const size_t src_size = strlen(src) + 1;
|
||||
const size_t max_dst_size = (size_t)LZ4_compressBound((int)src_size);
|
||||
int bytes_returned = 0;
|
||||
// Now build allocations for the data we'll be playing with.
|
||||
char *dst = (char*)calloc(1, max_dst_size);
|
||||
char *known_good_dst = (char*)calloc(1, max_dst_size);
|
||||
char *known_good_hc_dst = (char*)calloc(1, max_dst_size);
|
||||
if (dst == NULL || known_good_dst == NULL || known_good_hc_dst == NULL)
|
||||
run_screaming("Couldn't allocate memory for the destination buffers. Sad :(", 1);
|
||||
|
||||
// Create known-good buffers to verify our tests with other functions will produce the same results.
|
||||
bytes_returned = LZ4_compress_default(src, known_good_dst, (int)src_size, (int)max_dst_size);
|
||||
if (bytes_returned < 1)
|
||||
run_screaming("Couldn't create a known-good destination buffer for comparison... this is bad.", 1);
|
||||
const size_t src_comp_size = bytes_returned;
|
||||
bytes_returned = LZ4_compress_default(hc_src, known_good_hc_dst, (int)src_size, (int)max_dst_size);
|
||||
if (bytes_returned < 1)
|
||||
run_screaming("Couldn't create a known-good (highly compressible) destination buffer for comparison... this is bad.", 1);
|
||||
const size_t hc_src_comp_size = bytes_returned;
|
||||
|
||||
|
||||
/* LZ4_compress_default() */
|
||||
// This is the default function so we don't need to demonstrate how to use it. See basics.c if you need more basal information.
|
||||
|
||||
/* LZ4_compress_fast() */
|
||||
// Using this function is identical to LZ4_compress_default except we need to specify an "acceleration" value. Defaults to 1.
|
||||
memset(dst, 0, max_dst_size);
|
||||
bytes_returned = LZ4_compress_fast(src, dst, (int)src_size, (int)max_dst_size, 1);
|
||||
if (bytes_returned < 1)
|
||||
run_screaming("Failed to compress src using LZ4_compress_fast. echo $? for return code.", bytes_returned);
|
||||
if (memcmp(dst, known_good_dst, bytes_returned) != 0)
|
||||
run_screaming("According to memcmp(), the value we got in dst from LZ4_compress_fast doesn't match the known-good value. This is bad.", 1);
|
||||
|
||||
/* LZ4_compress_fast_extState() */
|
||||
// Using this function directly requires that we build an LZ4_stream_t struct ourselves. We do NOT have to reset it ourselves.
|
||||
memset(dst, 0, max_dst_size);
|
||||
LZ4_stream_t state;
|
||||
bytes_returned = LZ4_compress_fast_extState(&state, src, dst, (int)src_size, (int)max_dst_size, 1);
|
||||
if (bytes_returned < 1)
|
||||
run_screaming("Failed to compress src using LZ4_compress_fast_extState. echo $? for return code.", bytes_returned);
|
||||
if (memcmp(dst, known_good_dst, bytes_returned) != 0)
|
||||
run_screaming("According to memcmp(), the value we got in dst from LZ4_compress_fast_extState doesn't match the known-good value. This is bad.", 1);
|
||||
|
||||
/* LZ4_compress_generic */
|
||||
// When you can exactly control the inputs and options of your LZ4 needs, you can use LZ4_compress_generic and fixed (const)
|
||||
// values for the enum types such as dictionary and limitations. Any other direct-use is probably a bad idea.
|
||||
//
|
||||
// That said, the LZ4_compress_generic() function is 'static inline' and does not have a prototype in lz4.h to expose a symbol
|
||||
// for it. In other words: we can't access it directly. I don't want to submit a PR that modifies lz4.c/h. Yann and others can
|
||||
// do that if they feel it's worth expanding this example.
|
||||
//
|
||||
// I will, however, leave a skeleton of what would be required to use it directly:
|
||||
/*
|
||||
memset(dst, 0, max_dst_size);
|
||||
// LZ4_stream_t state: is already declared above. We can reuse it BUT we have to reset the stream ourselves between each call.
|
||||
LZ4_resetStream((LZ4_stream_t *)&state);
|
||||
// Since src size is small we know the following enums will be used: notLimited (0), byU16 (2), noDict (0), noDictIssue (0).
|
||||
bytes_returned = LZ4_compress_generic(&state, src, dst, src_size, max_dst_size, notLimited, byU16, noDict, noDictIssue, 1);
|
||||
if (bytes_returned < 1)
|
||||
run_screaming("Failed to compress src using LZ4_compress_generic. echo $? for return code.", bytes_returned);
|
||||
if (memcmp(dst, known_good_dst, bytes_returned) != 0)
|
||||
run_screaming("According to memcmp(), the value we got in dst from LZ4_compress_generic doesn't match the known-good value. This is bad.", 1);
|
||||
*/
|
||||
|
||||
|
||||
/* Benchmarking */
|
||||
/* Now we'll run a few rudimentary benchmarks with each function to demonstrate differences in speed based on the function used.
|
||||
* Remember, we cannot call LZ4_compress_generic() directly (yet) so it's disabled.
|
||||
*/
|
||||
// Suite A - Normal Compressibility
|
||||
char *dst_d = (char*)calloc(1, src_size);
|
||||
CHECK(dst_d!=NULL);
|
||||
memset(dst, 0, max_dst_size);
|
||||
printf("\nStarting suite A: Normal compressible text.\n");
|
||||
uint64_t time_taken__default = bench(known_good_dst, ID__LZ4_COMPRESS_DEFAULT, iterations, src, dst, src_size, max_dst_size, src_comp_size);
|
||||
uint64_t time_taken__fast = bench(known_good_dst, ID__LZ4_COMPRESS_FAST, iterations, src, dst, src_size, max_dst_size, src_comp_size);
|
||||
uint64_t time_taken__fast_extstate = bench(known_good_dst, ID__LZ4_COMPRESS_FAST_EXTSTATE, iterations, src, dst, src_size, max_dst_size, src_comp_size);
|
||||
//uint64_t time_taken__generic = bench(known_good_dst, ID__LZ4_COMPRESS_GENERIC, iterations, src, dst, src_size, max_dst_size, src_comp_size);
|
||||
uint64_t time_taken__decomp_safe = bench(src, ID__LZ4_DECOMPRESS_SAFE, iterations, known_good_dst, dst_d, src_size, max_dst_size, src_comp_size);
|
||||
uint64_t time_taken__decomp_fast = bench(src, ID__LZ4_DECOMPRESS_FAST, iterations, known_good_dst, dst_d, src_size, max_dst_size, src_comp_size);
|
||||
// Suite B - Highly Compressible
|
||||
memset(dst, 0, max_dst_size);
|
||||
printf("\nStarting suite B: Highly compressible text.\n");
|
||||
uint64_t time_taken_hc__default = bench(known_good_hc_dst, ID__LZ4_COMPRESS_DEFAULT, iterations, hc_src, dst, src_size, max_dst_size, hc_src_comp_size);
|
||||
uint64_t time_taken_hc__fast = bench(known_good_hc_dst, ID__LZ4_COMPRESS_FAST, iterations, hc_src, dst, src_size, max_dst_size, hc_src_comp_size);
|
||||
uint64_t time_taken_hc__fast_extstate = bench(known_good_hc_dst, ID__LZ4_COMPRESS_FAST_EXTSTATE, iterations, hc_src, dst, src_size, max_dst_size, hc_src_comp_size);
|
||||
//uint64_t time_taken_hc__generic = bench(known_good_hc_dst, ID__LZ4_COMPRESS_GENERIC, iterations, hc_src, dst, src_size, max_dst_size, hc_src_comp_size);
|
||||
uint64_t time_taken_hc__decomp_safe = bench(hc_src, ID__LZ4_DECOMPRESS_SAFE, iterations, known_good_hc_dst, dst_d, src_size, max_dst_size, hc_src_comp_size);
|
||||
uint64_t time_taken_hc__decomp_fast = bench(hc_src, ID__LZ4_DECOMPRESS_FAST, iterations, known_good_hc_dst, dst_d, src_size, max_dst_size, hc_src_comp_size);
|
||||
|
||||
// Report and leave.
|
||||
setlocale(LC_ALL, "");
|
||||
const char *format = "|%-14s|%-30s|%'14.9f|%'16d|%'14llu|%'13.2f%%|\n";
|
||||
const char *header_format = "|%-14s|%-30s|%14s|%16s|%14s|%14s|\n";
|
||||
const char *separator = "+--------------+------------------------------+--------------+----------------+--------------+--------------+\n";
|
||||
uint64_t iterllu = (uint64_t)iterations;
|
||||
printf("\n");
|
||||
printf("%s", separator);
|
||||
printf(header_format, "Source", "Function Benchmarked", "Total Seconds", "Iterations/sec", "ns/Iteration", "% of default");
|
||||
printf("%s", separator);
|
||||
printf(format, "Normal Text", "LZ4_compress_default()", (double)time_taken__default / BILLION, (int)(iterations / ((double)time_taken__default /BILLION)), time_taken__default / iterllu, (double)time_taken__default * 100 / time_taken__default);
|
||||
printf(format, "Normal Text", "LZ4_compress_fast()", (double)time_taken__fast / BILLION, (int)(iterations / ((double)time_taken__fast /BILLION)), time_taken__fast / iterllu, (double)time_taken__fast * 100 / time_taken__default);
|
||||
printf(format, "Normal Text", "LZ4_compress_fast_extState()", (double)time_taken__fast_extstate / BILLION, (int)(iterations / ((double)time_taken__fast_extstate /BILLION)), time_taken__fast_extstate / iterllu, (double)time_taken__fast_extstate * 100 / time_taken__default);
|
||||
//printf(format, "Normal Text", "LZ4_compress_generic()", (double)time_taken__generic / BILLION, (int)(iterations / ((double)time_taken__generic /BILLION)), (int)time_taken__generic / iterations, (double)time_taken__generic * 100 / time_taken__default);
|
||||
printf(format, "Normal Text", "LZ4_decompress_safe()", (double)time_taken__decomp_safe / BILLION, (int)(iterations / ((double)time_taken__decomp_safe /BILLION)), time_taken__decomp_safe / iterllu, (double)time_taken__decomp_safe * 100 / time_taken__default);
|
||||
printf(format, "Normal Text", "LZ4_decompress_fast()", (double)time_taken__decomp_fast / BILLION, (int)(iterations / ((double)time_taken__decomp_fast /BILLION)), time_taken__decomp_fast / iterllu, (double)time_taken__decomp_fast * 100 / time_taken__default);
|
||||
printf(header_format, "", "", "", "", "", "");
|
||||
printf(format, "Compressible", "LZ4_compress_default()", (double)time_taken_hc__default / BILLION, (int)(iterations / ((double)time_taken_hc__default /BILLION)), time_taken_hc__default / iterllu, (double)time_taken_hc__default * 100 / time_taken_hc__default);
|
||||
printf(format, "Compressible", "LZ4_compress_fast()", (double)time_taken_hc__fast / BILLION, (int)(iterations / ((double)time_taken_hc__fast /BILLION)), time_taken_hc__fast / iterllu, (double)time_taken_hc__fast * 100 / time_taken_hc__default);
|
||||
printf(format, "Compressible", "LZ4_compress_fast_extState()", (double)time_taken_hc__fast_extstate / BILLION, (int)(iterations / ((double)time_taken_hc__fast_extstate /BILLION)), time_taken_hc__fast_extstate / iterllu, (double)time_taken_hc__fast_extstate * 100 / time_taken_hc__default);
|
||||
//printf(format, "Compressible", "LZ4_compress_generic()", (double)time_taken_hc__generic / BILLION, (int)(iterations / ((double)time_taken_hc__generic /BILLION)), (int)time_taken_hc__generic / iterations, (double)time_taken_hc__generic * 100 / time_taken_hc__default);
|
||||
printf(format, "Compressible", "LZ4_decompress_safe()", (double)time_taken_hc__decomp_safe / BILLION, (int)(iterations / ((double)time_taken_hc__decomp_safe /BILLION)), time_taken_hc__decomp_safe / iterllu, (double)time_taken_hc__decomp_safe * 100 / time_taken_hc__default);
|
||||
printf(format, "Compressible", "LZ4_decompress_fast()", (double)time_taken_hc__decomp_fast / BILLION, (int)(iterations / ((double)time_taken_hc__decomp_fast /BILLION)), time_taken_hc__decomp_fast / iterllu, (double)time_taken_hc__decomp_fast * 100 / time_taken_hc__default);
|
||||
printf("%s", separator);
|
||||
printf("\n");
|
||||
printf("All done, ran %d iterations per test.\n", iterations);
|
||||
return 0;
|
||||
}
|
||||
209
expkg/vendor/lz4/examples/blockStreaming_doubleBuffer.c
vendored
Normal file
209
expkg/vendor/lz4/examples/blockStreaming_doubleBuffer.c
vendored
Normal file
@ -0,0 +1,209 @@
|
||||
// LZ4 streaming API example : double buffer
|
||||
// Copyright : Takayuki Matsuoka
|
||||
|
||||
|
||||
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
|
||||
# define _CRT_SECURE_NO_WARNINGS
|
||||
# define snprintf sprintf_s
|
||||
#endif
|
||||
#include "lz4.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <assert.h>
|
||||
|
||||
enum {
|
||||
BLOCK_BYTES = 1024 * 8,
|
||||
// BLOCK_BYTES = 1024 * 64,
|
||||
};
|
||||
|
||||
|
||||
size_t write_int(FILE* fp, int i) {
|
||||
return fwrite(&i, sizeof(i), 1, fp);
|
||||
}
|
||||
|
||||
size_t write_bin(FILE* fp, const void* array, size_t arrayBytes) {
|
||||
return fwrite(array, 1, arrayBytes, fp);
|
||||
}
|
||||
|
||||
size_t read_int(FILE* fp, int* i) {
|
||||
return fread(i, sizeof(*i), 1, fp);
|
||||
}
|
||||
|
||||
size_t read_bin(FILE* fp, void* array, size_t arrayBytes) {
|
||||
return fread(array, 1, arrayBytes, fp);
|
||||
}
|
||||
|
||||
|
||||
void test_compress(FILE* outFp, FILE* inpFp)
|
||||
{
|
||||
assert(outFp != NULL); assert(inpFp != NULL);
|
||||
|
||||
LZ4_stream_t lz4Stream_body;
|
||||
LZ4_stream_t* lz4Stream = &lz4Stream_body;
|
||||
|
||||
char inpBuf[2][BLOCK_BYTES];
|
||||
int inpBufIndex = 0;
|
||||
|
||||
LZ4_initStream(lz4Stream, sizeof (*lz4Stream));
|
||||
|
||||
for(;;) {
|
||||
char* const inpPtr = inpBuf[inpBufIndex];
|
||||
const int inpBytes = (int) read_bin(inpFp, inpPtr, BLOCK_BYTES);
|
||||
if(0 == inpBytes) {
|
||||
break;
|
||||
}
|
||||
|
||||
{
|
||||
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
|
||||
const int cmpBytes = LZ4_compress_fast_continue(
|
||||
lz4Stream, inpPtr, cmpBuf, inpBytes, sizeof(cmpBuf), 1);
|
||||
if(cmpBytes <= 0) {
|
||||
break;
|
||||
}
|
||||
write_int(outFp, cmpBytes);
|
||||
write_bin(outFp, cmpBuf, (size_t) cmpBytes);
|
||||
}
|
||||
|
||||
inpBufIndex = (inpBufIndex + 1) % 2;
|
||||
}
|
||||
|
||||
write_int(outFp, 0);
|
||||
}
|
||||
|
||||
|
||||
void test_decompress(FILE* outFp, FILE* inpFp)
|
||||
{
|
||||
assert(outFp != NULL); assert(inpFp != NULL);
|
||||
|
||||
LZ4_streamDecode_t lz4StreamDecode_body;
|
||||
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
|
||||
|
||||
char decBuf[2][BLOCK_BYTES];
|
||||
int decBufIndex = 0;
|
||||
|
||||
LZ4_setStreamDecode(lz4StreamDecode, NULL, 0);
|
||||
|
||||
for(;;) {
|
||||
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
|
||||
int cmpBytes = 0;
|
||||
|
||||
{
|
||||
const size_t readCount0 = read_int(inpFp, &cmpBytes);
|
||||
if(readCount0 != 1 || cmpBytes <= 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
const size_t readCount1 = read_bin(inpFp, cmpBuf, (size_t) cmpBytes);
|
||||
if(readCount1 != (size_t) cmpBytes) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
char* const decPtr = decBuf[decBufIndex];
|
||||
const int decBytes = LZ4_decompress_safe_continue(
|
||||
lz4StreamDecode, cmpBuf, decPtr, cmpBytes, BLOCK_BYTES);
|
||||
if(decBytes <= 0) {
|
||||
break;
|
||||
}
|
||||
write_bin(outFp, decPtr, (size_t) decBytes);
|
||||
}
|
||||
|
||||
decBufIndex = (decBufIndex + 1) % 2;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
int compare(FILE* fp0, FILE* fp1)
|
||||
{
|
||||
int result = 0;
|
||||
|
||||
assert(fp0 != NULL); assert(fp1 != NULL);
|
||||
|
||||
while(0 == result) {
|
||||
char b0[65536];
|
||||
char b1[65536];
|
||||
const size_t r0 = read_bin(fp0, b0, sizeof(b0));
|
||||
const size_t r1 = read_bin(fp1, b1, sizeof(b1));
|
||||
|
||||
result = (int) r0 - (int) r1;
|
||||
|
||||
if(0 == r0 || 0 == r1) {
|
||||
break;
|
||||
}
|
||||
if(0 == result) {
|
||||
result = memcmp(b0, b1, r0);
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
int main(int argc, char* argv[])
|
||||
{
|
||||
char inpFilename[256] = { 0 };
|
||||
char lz4Filename[256] = { 0 };
|
||||
char decFilename[256] = { 0 };
|
||||
|
||||
if(argc < 2) {
|
||||
printf("Please specify input filename\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
snprintf(inpFilename, 256, "%s", argv[1]);
|
||||
snprintf(lz4Filename, 256, "%s.lz4s-%d", argv[1], BLOCK_BYTES);
|
||||
snprintf(decFilename, 256, "%s.lz4s-%d.dec", argv[1], BLOCK_BYTES);
|
||||
|
||||
printf("inp = [%s]\n", inpFilename);
|
||||
printf("lz4 = [%s]\n", lz4Filename);
|
||||
printf("dec = [%s]\n", decFilename);
|
||||
|
||||
// compress
|
||||
{
|
||||
FILE* inpFp = fopen(inpFilename, "rb");
|
||||
FILE* outFp = fopen(lz4Filename, "wb");
|
||||
|
||||
printf("compress : %s -> %s\n", inpFilename, lz4Filename);
|
||||
test_compress(outFp, inpFp);
|
||||
printf("compress : done\n");
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
// decompress
|
||||
{
|
||||
FILE* inpFp = fopen(lz4Filename, "rb");
|
||||
FILE* outFp = fopen(decFilename, "wb");
|
||||
|
||||
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
|
||||
test_decompress(outFp, inpFp);
|
||||
printf("decompress : done\n");
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
// verify
|
||||
{
|
||||
FILE* inpFp = fopen(inpFilename, "rb");
|
||||
FILE* decFp = fopen(decFilename, "rb");
|
||||
|
||||
printf("verify : %s <-> %s\n", inpFilename, decFilename);
|
||||
const int cmp = compare(inpFp, decFp);
|
||||
if(0 == cmp) {
|
||||
printf("verify : OK\n");
|
||||
} else {
|
||||
printf("verify : NG\n");
|
||||
}
|
||||
|
||||
fclose(decFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
100
expkg/vendor/lz4/examples/blockStreaming_doubleBuffer.md
vendored
Normal file
100
expkg/vendor/lz4/examples/blockStreaming_doubleBuffer.md
vendored
Normal file
@ -0,0 +1,100 @@
|
||||
# LZ4 Streaming API Example : Double Buffer
|
||||
by *Takayuki Matsuoka*
|
||||
|
||||
`blockStreaming_doubleBuffer.c` is LZ4 Streaming API example which implements double buffer (de)compression.
|
||||
|
||||
Please note :
|
||||
|
||||
- Firstly, read "LZ4 Streaming API Basics".
|
||||
- This is relatively advanced application example.
|
||||
- Output file is not compatible with lz4frame and platform dependent.
|
||||
|
||||
|
||||
## What's the point of this example ?
|
||||
|
||||
- Handle huge file in small amount of memory
|
||||
- Always better compression ratio than Block API
|
||||
- Uniform block size
|
||||
|
||||
|
||||
## How the compression works
|
||||
|
||||
First of all, allocate "Double Buffer" for input and LZ4 compressed data buffer for output.
|
||||
Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
|
||||
|
||||
```
|
||||
Double Buffer
|
||||
|
||||
Page#1 Page#2
|
||||
+---------+---------+
|
||||
| Block#1 | |
|
||||
+----+----+---------+
|
||||
|
|
||||
v
|
||||
{Out#1}
|
||||
|
||||
|
||||
Prefix Dependency
|
||||
+---------+
|
||||
| |
|
||||
v |
|
||||
+---------+----+----+
|
||||
| Block#1 | Block#2 |
|
||||
+---------+----+----+
|
||||
|
|
||||
v
|
||||
{Out#2}
|
||||
|
||||
|
||||
External Dictionary Mode
|
||||
+---------+
|
||||
| |
|
||||
| v
|
||||
+----+----+---------+
|
||||
| Block#3 | Block#2 |
|
||||
+----+----+---------+
|
||||
|
|
||||
v
|
||||
{Out#3}
|
||||
|
||||
|
||||
Prefix Dependency
|
||||
+---------+
|
||||
| |
|
||||
v |
|
||||
+---------+----+----+
|
||||
| Block#3 | Block#4 |
|
||||
+---------+----+----+
|
||||
|
|
||||
v
|
||||
{Out#4}
|
||||
```
|
||||
|
||||
Next, read first block to double buffer's first page. And compress it by `LZ4_compress_continue()`.
|
||||
For the first time, LZ4 doesn't know any previous dependencies,
|
||||
so it just compress the line without dependencies and generates compressed block {Out#1} to LZ4 compressed data buffer.
|
||||
After that, write {Out#1} to the file.
|
||||
|
||||
Next, read second block to double buffer's second page. And compress it.
|
||||
This time, LZ4 can use dependency to Block#1 to improve compression ratio.
|
||||
This dependency is called "Prefix mode".
|
||||
|
||||
Next, read third block to double buffer's *first* page, and compress it.
|
||||
Also this time, LZ4 can use dependency to Block#2.
|
||||
This dependency is called "External Dictonaly mode".
|
||||
|
||||
Continue these procedure to the end of the file.
|
||||
|
||||
|
||||
## How the decompression works
|
||||
|
||||
Decompression will do reverse order.
|
||||
|
||||
- Read first compressed block.
|
||||
- Decompress it to the first page and write that page to the file.
|
||||
- Read second compressed block.
|
||||
- Decompress it to the second page and write that page to the file.
|
||||
- Read third compressed block.
|
||||
- Decompress it to the *first* page and write that page to the file.
|
||||
|
||||
Continue these procedure to the end of the compressed file.
|
||||
218
expkg/vendor/lz4/examples/blockStreaming_lineByLine.c
vendored
Normal file
218
expkg/vendor/lz4/examples/blockStreaming_lineByLine.c
vendored
Normal file
@ -0,0 +1,218 @@
|
||||
// LZ4 streaming API example : line-by-line logfile compression
|
||||
// by Takayuki Matsuoka
|
||||
|
||||
|
||||
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
|
||||
# define _CRT_SECURE_NO_WARNINGS
|
||||
# define snprintf sprintf_s
|
||||
#endif
|
||||
#include "lz4.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <assert.h>
|
||||
|
||||
static size_t write_uint16(FILE* fp, uint16_t i)
|
||||
{
|
||||
return fwrite(&i, sizeof(i), 1, fp);
|
||||
}
|
||||
|
||||
static size_t write_bin(FILE* fp, const void* array, int arrayBytes)
|
||||
{
|
||||
return fwrite(array, 1, arrayBytes, fp);
|
||||
}
|
||||
|
||||
static size_t read_uint16(FILE* fp, uint16_t* i)
|
||||
{
|
||||
return fread(i, sizeof(*i), 1, fp);
|
||||
}
|
||||
|
||||
static size_t read_bin(FILE* fp, void* array, int arrayBytes)
|
||||
{
|
||||
return fread(array, 1, arrayBytes, fp);
|
||||
}
|
||||
|
||||
|
||||
static void test_compress(
|
||||
FILE* outFp,
|
||||
FILE* inpFp,
|
||||
size_t messageMaxBytes,
|
||||
size_t ringBufferBytes)
|
||||
{
|
||||
assert(outFp != NULL); assert(inpFp != NULL);
|
||||
|
||||
LZ4_stream_t* const lz4Stream = LZ4_createStream();
|
||||
const size_t cmpBufBytes = LZ4_COMPRESSBOUND(messageMaxBytes);
|
||||
char* const cmpBuf = (char*) malloc(cmpBufBytes);
|
||||
char* const inpBuf = (char*) malloc(ringBufferBytes);
|
||||
int inpOffset = 0;
|
||||
|
||||
for ( ; ; )
|
||||
{
|
||||
char* const inpPtr = &inpBuf[inpOffset];
|
||||
|
||||
#if 0
|
||||
// Read random length data to the ring buffer.
|
||||
const int randomLength = (rand() % messageMaxBytes) + 1;
|
||||
const int inpBytes = (int) read_bin(inpFp, inpPtr, randomLength);
|
||||
if (0 == inpBytes) break;
|
||||
#else
|
||||
// Read line to the ring buffer.
|
||||
int inpBytes = 0;
|
||||
if (!fgets(inpPtr, (int) messageMaxBytes, inpFp))
|
||||
break;
|
||||
inpBytes = (int) strlen(inpPtr);
|
||||
#endif
|
||||
|
||||
{
|
||||
const int cmpBytes = LZ4_compress_fast_continue(
|
||||
lz4Stream, inpPtr, cmpBuf, inpBytes, (int) cmpBufBytes, 1);
|
||||
if (cmpBytes <= 0) break;
|
||||
write_uint16(outFp, (uint16_t) cmpBytes);
|
||||
write_bin(outFp, cmpBuf, cmpBytes);
|
||||
|
||||
// Add and wraparound the ringbuffer offset
|
||||
inpOffset += inpBytes;
|
||||
if ((size_t)inpOffset >= ringBufferBytes - messageMaxBytes) inpOffset = 0;
|
||||
}
|
||||
}
|
||||
write_uint16(outFp, 0);
|
||||
|
||||
free(inpBuf);
|
||||
free(cmpBuf);
|
||||
LZ4_freeStream(lz4Stream);
|
||||
}
|
||||
|
||||
|
||||
static void test_decompress(
|
||||
FILE* outFp,
|
||||
FILE* inpFp,
|
||||
size_t messageMaxBytes,
|
||||
size_t ringBufferBytes)
|
||||
{
|
||||
assert(outFp != NULL); assert(inpFp != NULL);
|
||||
|
||||
LZ4_streamDecode_t* const lz4StreamDecode = LZ4_createStreamDecode();
|
||||
char* const cmpBuf = (char*) malloc(LZ4_COMPRESSBOUND(messageMaxBytes));
|
||||
char* const decBuf = (char*) malloc(ringBufferBytes);
|
||||
int decOffset = 0;
|
||||
|
||||
for ( ; ; )
|
||||
{
|
||||
uint16_t cmpBytes = 0;
|
||||
|
||||
if (read_uint16(inpFp, &cmpBytes) != 1) break;
|
||||
if (cmpBytes == 0) break;
|
||||
if (read_bin(inpFp, cmpBuf, cmpBytes) != cmpBytes) break;
|
||||
|
||||
{
|
||||
char* const decPtr = &decBuf[decOffset];
|
||||
const int decBytes = LZ4_decompress_safe_continue(
|
||||
lz4StreamDecode, cmpBuf, decPtr, cmpBytes, (int) messageMaxBytes);
|
||||
if (decBytes <= 0) break;
|
||||
write_bin(outFp, decPtr, decBytes);
|
||||
|
||||
// Add and wraparound the ringbuffer offset
|
||||
decOffset += decBytes;
|
||||
if ((size_t)decOffset >= ringBufferBytes - messageMaxBytes) decOffset = 0;
|
||||
}
|
||||
}
|
||||
|
||||
free(decBuf);
|
||||
free(cmpBuf);
|
||||
LZ4_freeStreamDecode(lz4StreamDecode);
|
||||
}
|
||||
|
||||
|
||||
static int compare(FILE* f0, FILE* f1)
|
||||
{
|
||||
assert(f0 != NULL); assert(f1 != NULL);
|
||||
|
||||
int result = 0;
|
||||
const size_t tempBufferBytes = 65536;
|
||||
char* const b0 = (char*) malloc(tempBufferBytes);
|
||||
char* const b1 = (char*) malloc(tempBufferBytes);
|
||||
|
||||
while(0 == result)
|
||||
{
|
||||
const size_t r0 = fread(b0, 1, tempBufferBytes, f0);
|
||||
const size_t r1 = fread(b1, 1, tempBufferBytes, f1);
|
||||
|
||||
result = (int) r0 - (int) r1;
|
||||
|
||||
if (0 == r0 || 0 == r1) break;
|
||||
if (0 == result) result = memcmp(b0, b1, r0);
|
||||
}
|
||||
|
||||
free(b1);
|
||||
free(b0);
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
int main(int argc, char* argv[])
|
||||
{
|
||||
enum {
|
||||
MESSAGE_MAX_BYTES = 1024,
|
||||
RING_BUFFER_BYTES = 1024 * 256 + MESSAGE_MAX_BYTES,
|
||||
};
|
||||
|
||||
char inpFilename[256] = { 0 };
|
||||
char lz4Filename[256] = { 0 };
|
||||
char decFilename[256] = { 0 };
|
||||
|
||||
if (argc < 2)
|
||||
{
|
||||
printf("Please specify input filename\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
snprintf(inpFilename, 256, "%s", argv[1]);
|
||||
snprintf(lz4Filename, 256, "%s.lz4s", argv[1]);
|
||||
snprintf(decFilename, 256, "%s.lz4s.dec", argv[1]);
|
||||
|
||||
printf("inp = [%s]\n", inpFilename);
|
||||
printf("lz4 = [%s]\n", lz4Filename);
|
||||
printf("dec = [%s]\n", decFilename);
|
||||
|
||||
// compress
|
||||
{
|
||||
FILE* inpFp = fopen(inpFilename, "rb");
|
||||
FILE* outFp = fopen(lz4Filename, "wb");
|
||||
|
||||
test_compress(outFp, inpFp, MESSAGE_MAX_BYTES, RING_BUFFER_BYTES);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
// decompress
|
||||
{
|
||||
FILE* inpFp = fopen(lz4Filename, "rb");
|
||||
FILE* outFp = fopen(decFilename, "wb");
|
||||
|
||||
test_decompress(outFp, inpFp, MESSAGE_MAX_BYTES, RING_BUFFER_BYTES);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
// verify
|
||||
{
|
||||
FILE* inpFp = fopen(inpFilename, "rb");
|
||||
FILE* decFp = fopen(decFilename, "rb");
|
||||
|
||||
const int cmp = compare(inpFp, decFp);
|
||||
if (0 == cmp)
|
||||
printf("Verify : OK\n");
|
||||
else
|
||||
printf("Verify : NG\n");
|
||||
|
||||
fclose(decFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
122
expkg/vendor/lz4/examples/blockStreaming_lineByLine.md
vendored
Normal file
122
expkg/vendor/lz4/examples/blockStreaming_lineByLine.md
vendored
Normal file
@ -0,0 +1,122 @@
|
||||
# LZ4 Streaming API Example : Line by Line Text Compression
|
||||
by *Takayuki Matsuoka*
|
||||
|
||||
`blockStreaming_lineByLine.c` is LZ4 Streaming API example which implements line by line incremental (de)compression.
|
||||
|
||||
Please note the following restrictions :
|
||||
|
||||
- Firstly, read "LZ4 Streaming API Basics".
|
||||
- This is relatively advanced application example.
|
||||
- Output file is not compatible with lz4frame and platform dependent.
|
||||
|
||||
|
||||
## What's the point of this example ?
|
||||
|
||||
- Line by line incremental (de)compression.
|
||||
- Handle huge file in small amount of memory
|
||||
- Generally better compression ratio than Block API
|
||||
- Non-uniform block size
|
||||
|
||||
|
||||
## How the compression works
|
||||
|
||||
First of all, allocate "Ring Buffer" for input and LZ4 compressed data buffer for output.
|
||||
|
||||
```
|
||||
(1)
|
||||
Ring Buffer
|
||||
|
||||
+--------+
|
||||
| Line#1 |
|
||||
+---+----+
|
||||
|
|
||||
v
|
||||
{Out#1}
|
||||
|
||||
|
||||
(2)
|
||||
Prefix Mode Dependency
|
||||
+----+
|
||||
| |
|
||||
v |
|
||||
+--------+-+------+
|
||||
| Line#1 | Line#2 |
|
||||
+--------+---+----+
|
||||
|
|
||||
v
|
||||
{Out#2}
|
||||
|
||||
|
||||
(3)
|
||||
Prefix Prefix
|
||||
+----+ +----+
|
||||
| | | |
|
||||
v | v |
|
||||
+--------+-+------+-+------+
|
||||
| Line#1 | Line#2 | Line#3 |
|
||||
+--------+--------+---+----+
|
||||
|
|
||||
v
|
||||
{Out#3}
|
||||
|
||||
|
||||
(4)
|
||||
External Dictionary Mode
|
||||
+----+ +----+
|
||||
| | | |
|
||||
v | v |
|
||||
------+--------+-+------+-+--------+
|
||||
| .... | Line#X | Line#X+1 |
|
||||
------+--------+--------+-----+----+
|
||||
^ |
|
||||
| v
|
||||
| {Out#X+1}
|
||||
|
|
||||
Reset
|
||||
|
||||
|
||||
(5)
|
||||
Prefix
|
||||
+-----+
|
||||
| |
|
||||
v |
|
||||
------+--------+--------+----------+--+-------+
|
||||
| .... | Line#X | Line#X+1 | Line#X+2 |
|
||||
------+--------+--------+----------+-----+----+
|
||||
^ |
|
||||
| v
|
||||
| {Out#X+2}
|
||||
|
|
||||
Reset
|
||||
```
|
||||
|
||||
Next (see (1)), read first line to ringbuffer and compress it by `LZ4_compress_continue()`.
|
||||
For the first time, LZ4 doesn't know any previous dependencies,
|
||||
so it just compress the line without dependencies and generates compressed line {Out#1} to LZ4 compressed data buffer.
|
||||
After that, write {Out#1} to the file and forward ringbuffer offset.
|
||||
|
||||
Do the same things to second line (see (2)).
|
||||
But in this time, LZ4 can use dependency to Line#1 to improve compression ratio.
|
||||
This dependency is called "Prefix mode".
|
||||
|
||||
Eventually, we'll reach end of ringbuffer at Line#X (see (4)).
|
||||
This time, we should reset ringbuffer offset.
|
||||
After resetting, at Line#X+1 pointer is not adjacent, but LZ4 still maintain its memory.
|
||||
This is called "External Dictionary Mode".
|
||||
|
||||
In Line#X+2 (see (5)), finally LZ4 forget almost all memories but still remains Line#X+1.
|
||||
This is the same situation as Line#2.
|
||||
|
||||
Continue these procedures to the end of text file.
|
||||
|
||||
|
||||
## How the decompression works
|
||||
|
||||
Decompression will do reverse order.
|
||||
|
||||
- Read compressed line from the file to buffer.
|
||||
- Decompress it to the ringbuffer.
|
||||
- Output decompressed plain text line to the file.
|
||||
- Forward ringbuffer offset. If offset exceeds end of the ringbuffer, reset it.
|
||||
|
||||
Continue these procedures to the end of the compressed file.
|
||||
190
expkg/vendor/lz4/examples/blockStreaming_ringBuffer.c
vendored
Normal file
190
expkg/vendor/lz4/examples/blockStreaming_ringBuffer.c
vendored
Normal file
@ -0,0 +1,190 @@
|
||||
/* LZ4 streaming API example : ring buffer
|
||||
* Based on sample code from Takayuki Matsuoka */
|
||||
|
||||
|
||||
/**************************************
|
||||
* Compiler Options
|
||||
**************************************/
|
||||
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
|
||||
# define _CRT_SECURE_NO_WARNINGS
|
||||
# define snprintf sprintf_s
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
* Includes
|
||||
**************************************/
|
||||
#include <stdio.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include "lz4.h"
|
||||
|
||||
|
||||
enum {
|
||||
MESSAGE_MAX_BYTES = 1024,
|
||||
RING_BUFFER_BYTES = 1024 * 8 + MESSAGE_MAX_BYTES,
|
||||
DECODE_RING_BUFFER = RING_BUFFER_BYTES + MESSAGE_MAX_BYTES /* Intentionally larger, to test unsynchronized ring buffers */
|
||||
};
|
||||
|
||||
|
||||
size_t write_int32(FILE* fp, int32_t i) {
|
||||
return fwrite(&i, sizeof(i), 1, fp);
|
||||
}
|
||||
|
||||
size_t write_bin(FILE* fp, const void* array, int arrayBytes) {
|
||||
return fwrite(array, 1, arrayBytes, fp);
|
||||
}
|
||||
|
||||
size_t read_int32(FILE* fp, int32_t* i) {
|
||||
return fread(i, sizeof(*i), 1, fp);
|
||||
}
|
||||
|
||||
size_t read_bin(FILE* fp, void* array, int arrayBytes) {
|
||||
return fread(array, 1, arrayBytes, fp);
|
||||
}
|
||||
|
||||
|
||||
void test_compress(FILE* outFp, FILE* inpFp)
|
||||
{
|
||||
LZ4_stream_t lz4Stream_body = { { 0 } };
|
||||
LZ4_stream_t* lz4Stream = &lz4Stream_body;
|
||||
|
||||
static char inpBuf[RING_BUFFER_BYTES];
|
||||
int inpOffset = 0;
|
||||
|
||||
for(;;) {
|
||||
// Read random length ([1,MESSAGE_MAX_BYTES]) data to the ring buffer.
|
||||
char* const inpPtr = &inpBuf[inpOffset];
|
||||
const int randomLength = (rand() % MESSAGE_MAX_BYTES) + 1;
|
||||
const int inpBytes = (int) read_bin(inpFp, inpPtr, randomLength);
|
||||
if (0 == inpBytes) break;
|
||||
|
||||
{
|
||||
#define CMPBUFSIZE (LZ4_COMPRESSBOUND(MESSAGE_MAX_BYTES))
|
||||
char cmpBuf[CMPBUFSIZE];
|
||||
const int cmpBytes = LZ4_compress_fast_continue(lz4Stream, inpPtr, cmpBuf, inpBytes, CMPBUFSIZE, 0);
|
||||
if(cmpBytes <= 0) break;
|
||||
write_int32(outFp, cmpBytes);
|
||||
write_bin(outFp, cmpBuf, cmpBytes);
|
||||
|
||||
inpOffset += inpBytes;
|
||||
|
||||
// Wraparound the ringbuffer offset
|
||||
if(inpOffset >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES) inpOffset = 0;
|
||||
}
|
||||
}
|
||||
|
||||
write_int32(outFp, 0);
|
||||
}
|
||||
|
||||
|
||||
void test_decompress(FILE* outFp, FILE* inpFp)
|
||||
{
|
||||
static char decBuf[DECODE_RING_BUFFER];
|
||||
int decOffset = 0;
|
||||
LZ4_streamDecode_t lz4StreamDecode_body = { { 0 } };
|
||||
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
|
||||
|
||||
for(;;) {
|
||||
int cmpBytes = 0;
|
||||
char cmpBuf[CMPBUFSIZE];
|
||||
|
||||
{ const size_t r0 = read_int32(inpFp, &cmpBytes);
|
||||
if(r0 != 1 || cmpBytes <= 0) break;
|
||||
|
||||
const size_t r1 = read_bin(inpFp, cmpBuf, cmpBytes);
|
||||
if(r1 != (size_t) cmpBytes) break;
|
||||
}
|
||||
|
||||
{ char* const decPtr = &decBuf[decOffset];
|
||||
const int decBytes = LZ4_decompress_safe_continue(
|
||||
lz4StreamDecode, cmpBuf, decPtr, cmpBytes, MESSAGE_MAX_BYTES);
|
||||
if(decBytes <= 0) break;
|
||||
decOffset += decBytes;
|
||||
write_bin(outFp, decPtr, decBytes);
|
||||
|
||||
// Wraparound the ringbuffer offset
|
||||
if(decOffset >= DECODE_RING_BUFFER - MESSAGE_MAX_BYTES) decOffset = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
int compare(FILE* f0, FILE* f1)
|
||||
{
|
||||
int result = 0;
|
||||
|
||||
while (0 == result) {
|
||||
char b0[65536];
|
||||
char b1[65536];
|
||||
const size_t r0 = fread(b0, 1, sizeof(b0), f0);
|
||||
const size_t r1 = fread(b1, 1, sizeof(b1), f1);
|
||||
|
||||
result = (int) r0 - (int) r1;
|
||||
|
||||
if (0 == r0 || 0 == r1) break;
|
||||
|
||||
if (0 == result) result = memcmp(b0, b1, r0);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
int main(int argc, char** argv)
|
||||
{
|
||||
char inpFilename[256] = { 0 };
|
||||
char lz4Filename[256] = { 0 };
|
||||
char decFilename[256] = { 0 };
|
||||
|
||||
if (argc < 2) {
|
||||
printf("Please specify input filename\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
snprintf(inpFilename, 256, "%s", argv[1]);
|
||||
snprintf(lz4Filename, 256, "%s.lz4s-%d", argv[1], 0);
|
||||
snprintf(decFilename, 256, "%s.lz4s-%d.dec", argv[1], 0);
|
||||
|
||||
printf("inp = [%s]\n", inpFilename);
|
||||
printf("lz4 = [%s]\n", lz4Filename);
|
||||
printf("dec = [%s]\n", decFilename);
|
||||
|
||||
// compress
|
||||
{ FILE* const inpFp = fopen(inpFilename, "rb");
|
||||
FILE* const outFp = fopen(lz4Filename, "wb");
|
||||
|
||||
test_compress(outFp, inpFp);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
// decompress
|
||||
{ FILE* const inpFp = fopen(lz4Filename, "rb");
|
||||
FILE* const outFp = fopen(decFilename, "wb");
|
||||
|
||||
test_decompress(outFp, inpFp);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
// verify
|
||||
{ FILE* const inpFp = fopen(inpFilename, "rb");
|
||||
FILE* const decFp = fopen(decFilename, "rb");
|
||||
|
||||
const int cmp = compare(inpFp, decFp);
|
||||
if (0 == cmp) {
|
||||
printf("Verify : OK\n");
|
||||
} else {
|
||||
printf("Verify : NG\n");
|
||||
}
|
||||
|
||||
fclose(decFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
293
expkg/vendor/lz4/examples/dictionaryRandomAccess.c
vendored
Normal file
293
expkg/vendor/lz4/examples/dictionaryRandomAccess.c
vendored
Normal file
@ -0,0 +1,293 @@
|
||||
// LZ4 API example : Dictionary Random Access
|
||||
|
||||
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
|
||||
# define _CRT_SECURE_NO_WARNINGS
|
||||
# define snprintf sprintf_s
|
||||
#endif
|
||||
#include "lz4.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <assert.h>
|
||||
|
||||
#define MIN(x, y) ((x) < (y) ? (x) : (y))
|
||||
|
||||
enum {
|
||||
BLOCK_BYTES = 1024, /* 1 KiB of uncompressed data in a block */
|
||||
DICTIONARY_BYTES = 1024, /* Load a 1 KiB dictionary */
|
||||
MAX_BLOCKS = 1024 /* For simplicity of implementation */
|
||||
};
|
||||
|
||||
/**
|
||||
* Magic bytes for this test case.
|
||||
* This is not a great magic number because it is a common word in ASCII.
|
||||
* However, it is important to have some versioning system in your format.
|
||||
*/
|
||||
const char kTestMagic[] = { 'T', 'E', 'S', 'T' };
|
||||
|
||||
|
||||
void write_int(FILE* fp, int i) {
|
||||
size_t written = fwrite(&i, sizeof(i), 1, fp);
|
||||
if (written != 1) { exit(10); }
|
||||
}
|
||||
|
||||
void write_bin(FILE* fp, const void* array, size_t arrayBytes) {
|
||||
size_t written = fwrite(array, 1, arrayBytes, fp);
|
||||
if (written != arrayBytes) { exit(11); }
|
||||
}
|
||||
|
||||
void read_int(FILE* fp, int* i) {
|
||||
size_t read = fread(i, sizeof(*i), 1, fp);
|
||||
if (read != 1) { exit(12); }
|
||||
}
|
||||
|
||||
size_t read_bin(FILE* fp, void* array, size_t arrayBytes) {
|
||||
size_t read = fread(array, 1, arrayBytes, fp);
|
||||
if (ferror(fp)) { exit(12); }
|
||||
return read;
|
||||
}
|
||||
|
||||
void seek_bin(FILE* fp, long offset, int origin) {
|
||||
if (fseek(fp, offset, origin)) { exit(14); }
|
||||
}
|
||||
|
||||
|
||||
void test_compress(FILE* outFp, FILE* inpFp, void *dict, int dictSize)
|
||||
{
|
||||
assert(outFp != NULL); assert(inpFp != NULL);
|
||||
|
||||
LZ4_stream_t lz4Stream_body;
|
||||
LZ4_stream_t* lz4Stream = &lz4Stream_body;
|
||||
|
||||
char inpBuf[BLOCK_BYTES];
|
||||
int offsets[MAX_BLOCKS];
|
||||
int *offsetsEnd = offsets;
|
||||
|
||||
|
||||
LZ4_initStream(lz4Stream, sizeof(*lz4Stream));
|
||||
|
||||
/* Write header magic */
|
||||
write_bin(outFp, kTestMagic, sizeof(kTestMagic));
|
||||
|
||||
*offsetsEnd++ = sizeof(kTestMagic);
|
||||
/* Write compressed data blocks. Each block contains BLOCK_BYTES of plain
|
||||
data except possibly the last. */
|
||||
for(;;) {
|
||||
const int inpBytes = (int) read_bin(inpFp, inpBuf, BLOCK_BYTES);
|
||||
if(0 == inpBytes) {
|
||||
break;
|
||||
}
|
||||
|
||||
/* Forget previously compressed data and load the dictionary */
|
||||
LZ4_loadDict(lz4Stream, (const char*) dict, dictSize);
|
||||
{
|
||||
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
|
||||
const int cmpBytes = LZ4_compress_fast_continue(
|
||||
lz4Stream, inpBuf, cmpBuf, inpBytes, sizeof(cmpBuf), 1);
|
||||
if(cmpBytes <= 0) { exit(1); }
|
||||
write_bin(outFp, cmpBuf, (size_t)cmpBytes);
|
||||
/* Keep track of the offsets */
|
||||
*offsetsEnd = *(offsetsEnd - 1) + cmpBytes;
|
||||
++offsetsEnd;
|
||||
}
|
||||
if (offsetsEnd - offsets > MAX_BLOCKS) { exit(2); }
|
||||
}
|
||||
/* Write the tailing jump table */
|
||||
{
|
||||
int *ptr = offsets;
|
||||
while (ptr != offsetsEnd) {
|
||||
write_int(outFp, *ptr++);
|
||||
}
|
||||
write_int(outFp, (int) (offsetsEnd - offsets));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
void test_decompress(FILE* outFp, FILE* inpFp, void *dict, int dictSize, int offset, int length)
|
||||
{
|
||||
assert(outFp != NULL); assert(inpFp != NULL);
|
||||
|
||||
LZ4_streamDecode_t lz4StreamDecode_body;
|
||||
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
|
||||
|
||||
/* The blocks [currentBlock, endBlock) contain the data we want */
|
||||
int currentBlock = offset / BLOCK_BYTES;
|
||||
int endBlock = ((offset + length - 1) / BLOCK_BYTES) + 1;
|
||||
|
||||
char decBuf[BLOCK_BYTES];
|
||||
int offsets[MAX_BLOCKS];
|
||||
|
||||
/* Special cases */
|
||||
if (length == 0) { return; }
|
||||
|
||||
/* Read the magic bytes */
|
||||
{
|
||||
char magic[sizeof(kTestMagic)];
|
||||
size_t read = read_bin(inpFp, magic, sizeof(magic));
|
||||
if (read != sizeof(magic)) { exit(1); }
|
||||
if (memcmp(kTestMagic, magic, sizeof(magic))) { exit(2); }
|
||||
}
|
||||
|
||||
/* Read the offsets tail */
|
||||
{
|
||||
int numOffsets;
|
||||
int block;
|
||||
int *offsetsPtr = offsets;
|
||||
seek_bin(inpFp, -4, SEEK_END);
|
||||
read_int(inpFp, &numOffsets);
|
||||
if (numOffsets <= endBlock) { exit(3); }
|
||||
seek_bin(inpFp, -4 * (numOffsets + 1), SEEK_END);
|
||||
for (block = 0; block <= endBlock; ++block) {
|
||||
read_int(inpFp, offsetsPtr++);
|
||||
}
|
||||
}
|
||||
/* Seek to the first block to read */
|
||||
seek_bin(inpFp, offsets[currentBlock], SEEK_SET);
|
||||
offset = offset % BLOCK_BYTES;
|
||||
|
||||
/* Start decoding */
|
||||
for(; currentBlock < endBlock; ++currentBlock) {
|
||||
char cmpBuf[LZ4_COMPRESSBOUND(BLOCK_BYTES)];
|
||||
/* The difference in offsets is the size of the block */
|
||||
int cmpBytes = offsets[currentBlock + 1] - offsets[currentBlock];
|
||||
{
|
||||
const size_t read = read_bin(inpFp, cmpBuf, (size_t)cmpBytes);
|
||||
if(read != (size_t)cmpBytes) { exit(4); }
|
||||
}
|
||||
|
||||
/* Load the dictionary */
|
||||
LZ4_setStreamDecode(lz4StreamDecode, (const char*) dict, dictSize);
|
||||
{
|
||||
const int decBytes = LZ4_decompress_safe_continue(
|
||||
lz4StreamDecode, cmpBuf, decBuf, cmpBytes, BLOCK_BYTES);
|
||||
if(decBytes <= 0) { exit(5); }
|
||||
{
|
||||
/* Write out the part of the data we care about */
|
||||
int blockLength = MIN(length, (decBytes - offset));
|
||||
write_bin(outFp, decBuf + offset, (size_t)blockLength);
|
||||
offset = 0;
|
||||
length -= blockLength;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
int compare(FILE* fp0, FILE* fp1, int length)
|
||||
{
|
||||
int result = 0;
|
||||
|
||||
assert(fp0 != NULL); assert(fp1 != NULL);
|
||||
|
||||
while(0 == result) {
|
||||
char b0[4096];
|
||||
char b1[4096];
|
||||
const size_t r0 = read_bin(fp0, b0, MIN(length, (int)sizeof(b0)));
|
||||
const size_t r1 = read_bin(fp1, b1, MIN(length, (int)sizeof(b1)));
|
||||
|
||||
result = (int) r0 - (int) r1;
|
||||
|
||||
if(0 == r0 || 0 == r1) {
|
||||
break;
|
||||
}
|
||||
if(0 == result) {
|
||||
result = memcmp(b0, b1, r0);
|
||||
}
|
||||
length -= r0;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
int main(int argc, char* argv[])
|
||||
{
|
||||
char inpFilename[256] = { 0 };
|
||||
char lz4Filename[256] = { 0 };
|
||||
char decFilename[256] = { 0 };
|
||||
char dictFilename[256] = { 0 };
|
||||
int offset;
|
||||
int length;
|
||||
char dict[DICTIONARY_BYTES];
|
||||
int dictSize;
|
||||
|
||||
if(argc < 5) {
|
||||
printf("Usage: %s input dictionary offset length", argv[0]);
|
||||
return 0;
|
||||
}
|
||||
|
||||
snprintf(inpFilename, 256, "%s", argv[1]);
|
||||
snprintf(lz4Filename, 256, "%s.lz4s-%d", argv[1], BLOCK_BYTES);
|
||||
snprintf(decFilename, 256, "%s.lz4s-%d.dec", argv[1], BLOCK_BYTES);
|
||||
snprintf(dictFilename, 256, "%s", argv[2]);
|
||||
offset = atoi(argv[3]);
|
||||
length = atoi(argv[4]);
|
||||
|
||||
printf("inp = [%s]\n", inpFilename);
|
||||
printf("lz4 = [%s]\n", lz4Filename);
|
||||
printf("dec = [%s]\n", decFilename);
|
||||
printf("dict = [%s]\n", dictFilename);
|
||||
printf("offset = [%d]\n", offset);
|
||||
printf("length = [%d]\n", length);
|
||||
|
||||
assert(offset < INT32_MAX); assert(length < INT32_MAX);
|
||||
|
||||
/* Load dictionary */
|
||||
{
|
||||
FILE* dictFp = fopen(dictFilename, "rb");
|
||||
|
||||
assert(dictFp != NULL);
|
||||
dictSize = (int)read_bin(dictFp, dict, DICTIONARY_BYTES);
|
||||
fclose(dictFp);
|
||||
}
|
||||
|
||||
/* compress */
|
||||
{
|
||||
FILE* inpFp = fopen(inpFilename, "rb");
|
||||
FILE* outFp = fopen(lz4Filename, "wb");
|
||||
|
||||
printf("compress : %s -> %s\n", inpFilename, lz4Filename);
|
||||
test_compress(outFp, inpFp, dict, dictSize);
|
||||
printf("compress : done\n");
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
/* decompress */
|
||||
{
|
||||
FILE* inpFp = fopen(lz4Filename, "rb");
|
||||
FILE* outFp = fopen(decFilename, "wb");
|
||||
|
||||
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
|
||||
test_decompress(outFp, inpFp, dict, dictSize, offset, length);
|
||||
printf("decompress : done\n");
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
/* verify */
|
||||
{
|
||||
FILE* inpFp = fopen(inpFilename, "rb");
|
||||
FILE* decFp = fopen(decFilename, "rb");
|
||||
|
||||
assert(inpFp != NULL);assert(decFp != NULL);
|
||||
seek_bin(inpFp, offset, SEEK_SET);
|
||||
|
||||
printf("verify : %s <-> %s\n", inpFilename, decFilename);
|
||||
const int cmp = compare(inpFp, decFp, length);
|
||||
if(0 == cmp) {
|
||||
printf("verify : OK\n");
|
||||
} else {
|
||||
printf("verify : NG\n");
|
||||
}
|
||||
|
||||
fclose(decFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
67
expkg/vendor/lz4/examples/dictionaryRandomAccess.md
vendored
Normal file
67
expkg/vendor/lz4/examples/dictionaryRandomAccess.md
vendored
Normal file
@ -0,0 +1,67 @@
|
||||
# LZ4 API Example : Dictionary Random Access
|
||||
|
||||
`dictionaryRandomAccess.c` is LZ4 API example which implements dictionary compression and random access decompression.
|
||||
|
||||
Please note that the output file is not compatible with lz4frame and is platform dependent.
|
||||
|
||||
|
||||
## What's the point of this example ?
|
||||
|
||||
- Dictionary based compression for homogeneous files.
|
||||
- Random access to compressed blocks.
|
||||
|
||||
|
||||
## How the compression works
|
||||
|
||||
Reads the dictionary from a file, and uses it as the history for each block.
|
||||
This allows each block to be independent, but maintains compression ratio.
|
||||
|
||||
```
|
||||
Dictionary
|
||||
+
|
||||
|
|
||||
v
|
||||
+---------+
|
||||
| Block#1 |
|
||||
+----+----+
|
||||
|
|
||||
v
|
||||
{Out#1}
|
||||
|
||||
|
||||
Dictionary
|
||||
+
|
||||
|
|
||||
v
|
||||
+---------+
|
||||
| Block#2 |
|
||||
+----+----+
|
||||
|
|
||||
v
|
||||
{Out#2}
|
||||
```
|
||||
|
||||
After writing the magic bytes `TEST` and then the compressed blocks, write out the jump table.
|
||||
The last 4 bytes is an integer containing the number of blocks in the stream.
|
||||
If there are `N` blocks, then just before the last 4 bytes is `N + 1` 4 byte integers containing the offsets at the beginning and end of each block.
|
||||
Let `Offset#K` be the total number of bytes written after writing out `Block#K` *including* the magic bytes for simplicity.
|
||||
|
||||
```
|
||||
+------+---------+ +---------+---+----------+ +----------+-----+
|
||||
| TEST | Block#1 | ... | Block#N | 4 | Offset#1 | ... | Offset#N | N+1 |
|
||||
+------+---------+ +---------+---+----------+ +----------+-----+
|
||||
```
|
||||
|
||||
## How the decompression works
|
||||
|
||||
Decompression will do reverse order.
|
||||
|
||||
- Seek to the last 4 bytes of the file and read the number of offsets.
|
||||
- Read each offset into an array.
|
||||
- Seek to the first block containing data we want to read.
|
||||
We know where to look because we know each block contains a fixed amount of uncompressed data, except possibly the last.
|
||||
- Decompress it and write what data we need from it to the file.
|
||||
- Read the next block.
|
||||
- Decompress it and write that page to the file.
|
||||
|
||||
Continue these procedures until all the required data has been read.
|
||||
241
expkg/vendor/lz4/examples/fileCompress.c
vendored
Normal file
241
expkg/vendor/lz4/examples/fileCompress.c
vendored
Normal file
@ -0,0 +1,241 @@
|
||||
/* LZ4file API example : compress a file
|
||||
* Modified from an example code by anjiahao
|
||||
*
|
||||
* This example will demonstrate how
|
||||
* to manipulate lz4 compressed files like
|
||||
* normal files */
|
||||
|
||||
#include <assert.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
|
||||
#include <lz4file.h>
|
||||
|
||||
|
||||
#define CHUNK_SIZE (16*1024)
|
||||
|
||||
static size_t get_file_size(char *filename)
|
||||
{
|
||||
struct stat statbuf;
|
||||
|
||||
if (filename == NULL) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
if(stat(filename,&statbuf)) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
return (size_t)statbuf.st_size;
|
||||
}
|
||||
|
||||
static int compress_file(FILE* f_in, FILE* f_out)
|
||||
{
|
||||
assert(f_in != NULL); assert(f_out != NULL);
|
||||
|
||||
LZ4F_errorCode_t ret = LZ4F_OK_NoError;
|
||||
LZ4_writeFile_t* lz4fWrite;
|
||||
void* const buf = malloc(CHUNK_SIZE);
|
||||
if (!buf) {
|
||||
printf("error: memory allocation failed \n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Of course, you can also use prefsPtr to
|
||||
* set the parameters of the compressed file
|
||||
* NULL is use default
|
||||
*/
|
||||
ret = LZ4F_writeOpen(&lz4fWrite, f_out, NULL);
|
||||
if (LZ4F_isError(ret)) {
|
||||
printf("LZ4F_writeOpen error: %s\n", LZ4F_getErrorName(ret));
|
||||
free(buf);
|
||||
return 1;
|
||||
}
|
||||
|
||||
while (1) {
|
||||
size_t len = fread(buf, 1, CHUNK_SIZE, f_in);
|
||||
|
||||
if (ferror(f_in)) {
|
||||
printf("fread error\n");
|
||||
goto out;
|
||||
}
|
||||
|
||||
/* nothing to read */
|
||||
if (len == 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
ret = LZ4F_write(lz4fWrite, buf, len);
|
||||
if (LZ4F_isError(ret)) {
|
||||
printf("LZ4F_write: %s\n", LZ4F_getErrorName(ret));
|
||||
goto out;
|
||||
}
|
||||
}
|
||||
|
||||
out:
|
||||
free(buf);
|
||||
if (LZ4F_isError(LZ4F_writeClose(lz4fWrite))) {
|
||||
printf("LZ4F_writeClose: %s\n", LZ4F_getErrorName(ret));
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int decompress_file(FILE* f_in, FILE* f_out)
|
||||
{
|
||||
assert(f_in != NULL); assert(f_out != NULL);
|
||||
|
||||
LZ4F_errorCode_t ret = LZ4F_OK_NoError;
|
||||
LZ4_readFile_t* lz4fRead;
|
||||
void* const buf= malloc(CHUNK_SIZE);
|
||||
if (!buf) {
|
||||
printf("error: memory allocation failed \n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
ret = LZ4F_readOpen(&lz4fRead, f_in);
|
||||
if (LZ4F_isError(ret)) {
|
||||
printf("LZ4F_readOpen error: %s\n", LZ4F_getErrorName(ret));
|
||||
free(buf);
|
||||
return 1;
|
||||
}
|
||||
|
||||
while (1) {
|
||||
ret = LZ4F_read(lz4fRead, buf, CHUNK_SIZE);
|
||||
if (LZ4F_isError(ret)) {
|
||||
printf("LZ4F_read error: %s\n", LZ4F_getErrorName(ret));
|
||||
goto out;
|
||||
}
|
||||
|
||||
/* nothing to read */
|
||||
if (ret == 0) {
|
||||
break;
|
||||
}
|
||||
|
||||
if(fwrite(buf, 1, ret, f_out) != ret) {
|
||||
printf("write error!\n");
|
||||
goto out;
|
||||
}
|
||||
}
|
||||
|
||||
out:
|
||||
free(buf);
|
||||
if (LZ4F_isError(LZ4F_readClose(lz4fRead))) {
|
||||
printf("LZ4F_readClose: %s\n", LZ4F_getErrorName(ret));
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (ret) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
/* @return 0 if both FILE* have identical content */
|
||||
int compareFiles(FILE* fp0, FILE* fp1)
|
||||
{
|
||||
int result = 0;
|
||||
assert(fp0 != NULL); assert(fp1 != NULL);
|
||||
|
||||
while (result==0) {
|
||||
char b0[1024];
|
||||
char b1[1024];
|
||||
size_t const r0 = fread(b0, 1, sizeof(b0), fp0);
|
||||
size_t const r1 = fread(b1, 1, sizeof(b1), fp1);
|
||||
|
||||
result = (r0 != r1);
|
||||
if (!r0 || !r1) break;
|
||||
if (r0 == r1) result = memcmp(b0, b1, r0);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
int main(int argc, const char **argv) {
|
||||
char inpFilename[256] = { 0 };
|
||||
char lz4Filename[256] = { 0 };
|
||||
char decFilename[256] = { 0 };
|
||||
|
||||
if (argc < 2) {
|
||||
printf("Please specify input filename\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
snprintf(inpFilename, 256, "%s", argv[1]);
|
||||
snprintf(lz4Filename, 256, "%s.lz4", argv[1]);
|
||||
snprintf(decFilename, 256, "%s.lz4.dec", argv[1]);
|
||||
|
||||
printf("inp = [%s]\n", inpFilename);
|
||||
printf("lz4 = [%s]\n", lz4Filename);
|
||||
printf("dec = [%s]\n", decFilename);
|
||||
|
||||
/* compress */
|
||||
{ FILE* const inpFp = fopen(inpFilename, "rb");
|
||||
FILE* const outFp = fopen(lz4Filename, "wb");
|
||||
printf("compress : %s -> %s\n", inpFilename, lz4Filename);
|
||||
int ret = compress_file(inpFp, outFp);
|
||||
fclose(inpFp);
|
||||
fclose(outFp);
|
||||
|
||||
if (ret) {
|
||||
printf("compression error: %s\n", LZ4F_getErrorName(ret));
|
||||
return 1;
|
||||
}
|
||||
|
||||
printf("%s: %zu → %zu bytes, %.1f%%\n",
|
||||
inpFilename,
|
||||
get_file_size(inpFilename),
|
||||
get_file_size(lz4Filename), /* might overflow is size_t is 32 bits and size_{in,out} > 4 GB */
|
||||
(double)get_file_size(lz4Filename) / (double)get_file_size(inpFilename) * 100);
|
||||
|
||||
printf("compress : done\n");
|
||||
}
|
||||
|
||||
/* decompress */
|
||||
{
|
||||
FILE* const inpFp = fopen(lz4Filename, "rb");
|
||||
FILE* const outFp = fopen(decFilename, "wb");
|
||||
|
||||
if (inpFp == NULL) exit(1);
|
||||
if (outFp == NULL) exit(1);
|
||||
|
||||
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
|
||||
int ret = decompress_file(inpFp, outFp);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
|
||||
if (ret) {
|
||||
printf("compression error: %s\n", LZ4F_getErrorName(ret));
|
||||
return 1;
|
||||
}
|
||||
|
||||
printf("decompress : done\n");
|
||||
}
|
||||
|
||||
/* verify */
|
||||
{ FILE* const inpFp = fopen(inpFilename, "rb");
|
||||
FILE* const decFp = fopen(decFilename, "rb");
|
||||
|
||||
if (inpFp == NULL) exit(1);
|
||||
if (decFp == NULL) exit(1);
|
||||
|
||||
printf("verify : %s <-> %s\n", inpFilename, decFilename);
|
||||
int const cmp = compareFiles(inpFp, decFp);
|
||||
|
||||
fclose(decFp);
|
||||
fclose(inpFp);
|
||||
|
||||
if (cmp) {
|
||||
printf("corruption detected : decompressed file differs from original\n");
|
||||
return cmp;
|
||||
}
|
||||
|
||||
printf("verify : OK\n");
|
||||
}
|
||||
|
||||
}
|
||||
489
expkg/vendor/lz4/examples/frameCompress.c
vendored
Normal file
489
expkg/vendor/lz4/examples/frameCompress.c
vendored
Normal file
@ -0,0 +1,489 @@
|
||||
/* LZ4frame API example : compress a file
|
||||
* Modified from an example code by Zbigniew Jędrzejewski-Szmek
|
||||
*
|
||||
* This example streams an input file into an output file
|
||||
* using a bounded memory budget.
|
||||
* Input is read in chunks of IN_CHUNK_SIZE */
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <errno.h>
|
||||
#include <assert.h>
|
||||
|
||||
#include <getopt.h>
|
||||
#include <lz4frame.h>
|
||||
#include <lz4frame_static.h>
|
||||
|
||||
#define IN_CHUNK_SIZE (16*1024)
|
||||
|
||||
static const LZ4F_preferences_t kPrefs = {
|
||||
{ LZ4F_max256KB, LZ4F_blockLinked, LZ4F_noContentChecksum, LZ4F_frame,
|
||||
0 /* unknown content size */, 0 /* no dictID */ , LZ4F_noBlockChecksum },
|
||||
0, /* compression level; 0 == default */
|
||||
0, /* autoflush */
|
||||
0, /* favor decompression speed */
|
||||
{ 0, 0, 0 }, /* reserved, must be set to 0 */
|
||||
};
|
||||
|
||||
|
||||
/* safe_fwrite() :
|
||||
* performs fwrite(), ensure operation success, or immediately exit() */
|
||||
static void safe_fwrite(void* buf, size_t eltSize, size_t nbElt, FILE* f)
|
||||
{
|
||||
size_t const writtenSize = fwrite(buf, eltSize, nbElt, f);
|
||||
size_t const expectedSize = eltSize * nbElt;
|
||||
if (nbElt>0) assert(expectedSize / nbElt == eltSize); /* check overflow */
|
||||
if (writtenSize < expectedSize) {
|
||||
if (ferror(f)) /* note : ferror() must follow fwrite */
|
||||
fprintf(stderr, "Write failed \n");
|
||||
else
|
||||
fprintf(stderr, "Write too short \n");
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/* ================================================= */
|
||||
/* Streaming Compression example */
|
||||
/* ================================================= */
|
||||
|
||||
typedef struct {
|
||||
int error;
|
||||
unsigned long long size_in;
|
||||
unsigned long long size_out;
|
||||
} compressResult_t;
|
||||
|
||||
static compressResult_t
|
||||
compress_file_internal(FILE* f_in, FILE* f_out,
|
||||
LZ4F_compressionContext_t ctx,
|
||||
void* inBuff, size_t inChunkSize,
|
||||
void* outBuff, size_t outCapacity,
|
||||
FILE* f_unc, long uncOffset)
|
||||
{
|
||||
compressResult_t result = { 1, 0, 0 }; /* result for an error */
|
||||
long long count_in = 0, count_out, bytesToOffset = -1;
|
||||
|
||||
assert(f_in != NULL); assert(f_out != NULL);
|
||||
assert(ctx != NULL);
|
||||
assert(outCapacity >= LZ4F_HEADER_SIZE_MAX);
|
||||
assert(outCapacity >= LZ4F_compressBound(inChunkSize, &kPrefs));
|
||||
|
||||
/* write frame header */
|
||||
{ size_t const headerSize = LZ4F_compressBegin(ctx, outBuff, outCapacity, &kPrefs);
|
||||
if (LZ4F_isError(headerSize)) {
|
||||
printf("Failed to start compression: error %u \n", (unsigned)headerSize);
|
||||
return result;
|
||||
}
|
||||
count_out = headerSize;
|
||||
printf("Buffer size is %u bytes, header size %u bytes \n",
|
||||
(unsigned)outCapacity, (unsigned)headerSize);
|
||||
safe_fwrite(outBuff, 1, headerSize, f_out);
|
||||
}
|
||||
|
||||
/* stream file */
|
||||
for (;;) {
|
||||
size_t compressedSize;
|
||||
long long inSize = IN_CHUNK_SIZE;
|
||||
if (uncOffset >= 0) {
|
||||
bytesToOffset = uncOffset - count_in;
|
||||
|
||||
/* read only remaining bytes to offset position */
|
||||
if (bytesToOffset < IN_CHUNK_SIZE && bytesToOffset > 0) {
|
||||
inSize = bytesToOffset;
|
||||
}
|
||||
}
|
||||
|
||||
/* input data is at uncompressed data offset */
|
||||
if (bytesToOffset <= 0 && uncOffset >= 0 && f_unc) {
|
||||
size_t const readSize = fread(inBuff, 1, inSize, f_unc);
|
||||
if (readSize == 0) {
|
||||
uncOffset = -1;
|
||||
continue;
|
||||
}
|
||||
count_in += readSize;
|
||||
compressedSize = LZ4F_uncompressedUpdate(ctx,
|
||||
outBuff, outCapacity,
|
||||
inBuff, readSize,
|
||||
NULL);
|
||||
} else {
|
||||
size_t const readSize = fread(inBuff, 1, inSize, f_in);
|
||||
if (readSize == 0) break; /* nothing left to read from input file */
|
||||
count_in += readSize;
|
||||
compressedSize = LZ4F_compressUpdate(ctx,
|
||||
outBuff, outCapacity,
|
||||
inBuff, readSize,
|
||||
NULL);
|
||||
|
||||
}
|
||||
|
||||
if (LZ4F_isError(compressedSize)) {
|
||||
printf("Compression failed: error %u \n", (unsigned)compressedSize);
|
||||
return result;
|
||||
}
|
||||
|
||||
printf("Writing %u bytes\n", (unsigned)compressedSize);
|
||||
safe_fwrite(outBuff, 1, compressedSize, f_out);
|
||||
count_out += compressedSize;
|
||||
}
|
||||
|
||||
/* flush whatever remains within internal buffers */
|
||||
{ size_t const compressedSize = LZ4F_compressEnd(ctx,
|
||||
outBuff, outCapacity,
|
||||
NULL);
|
||||
if (LZ4F_isError(compressedSize)) {
|
||||
printf("Failed to end compression: error %u \n", (unsigned)compressedSize);
|
||||
return result;
|
||||
}
|
||||
|
||||
printf("Writing %u bytes \n", (unsigned)compressedSize);
|
||||
safe_fwrite(outBuff, 1, compressedSize, f_out);
|
||||
count_out += compressedSize;
|
||||
}
|
||||
|
||||
result.size_in = count_in;
|
||||
result.size_out = count_out;
|
||||
result.error = 0;
|
||||
return result;
|
||||
}
|
||||
|
||||
static compressResult_t
|
||||
compress_file(FILE* f_in, FILE* f_out,
|
||||
FILE* f_unc, int uncOffset)
|
||||
{
|
||||
assert(f_in != NULL);
|
||||
assert(f_out != NULL);
|
||||
|
||||
/* resource allocation */
|
||||
LZ4F_compressionContext_t ctx;
|
||||
size_t const ctxCreation = LZ4F_createCompressionContext(&ctx, LZ4F_VERSION);
|
||||
void* const src = malloc(IN_CHUNK_SIZE);
|
||||
size_t const outbufCapacity = LZ4F_compressBound(IN_CHUNK_SIZE, &kPrefs); /* large enough for any input <= IN_CHUNK_SIZE */
|
||||
void* const outbuff = malloc(outbufCapacity);
|
||||
|
||||
compressResult_t result = { 1, 0, 0 }; /* == error (default) */
|
||||
if (!LZ4F_isError(ctxCreation) && src && outbuff) {
|
||||
result = compress_file_internal(f_in, f_out,
|
||||
ctx,
|
||||
src, IN_CHUNK_SIZE,
|
||||
outbuff, outbufCapacity,
|
||||
f_unc, uncOffset);
|
||||
} else {
|
||||
printf("error : resource allocation failed \n");
|
||||
}
|
||||
|
||||
LZ4F_freeCompressionContext(ctx); /* supports free on NULL */
|
||||
free(src);
|
||||
free(outbuff);
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
/* ================================================= */
|
||||
/* Streaming decompression example */
|
||||
/* ================================================= */
|
||||
|
||||
static size_t get_block_size(const LZ4F_frameInfo_t* info) {
|
||||
switch (info->blockSizeID) {
|
||||
case LZ4F_default:
|
||||
case LZ4F_max64KB: return 1 << 16;
|
||||
case LZ4F_max256KB: return 1 << 18;
|
||||
case LZ4F_max1MB: return 1 << 20;
|
||||
case LZ4F_max4MB: return 1 << 22;
|
||||
default:
|
||||
printf("Impossible with expected frame specification (<=v1.6.1)\n");
|
||||
exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
/* @return : 1==error, 0==success */
|
||||
static int
|
||||
decompress_file_internal(FILE* f_in, FILE* f_out,
|
||||
LZ4F_dctx* dctx,
|
||||
void* src, size_t srcCapacity, size_t filled, size_t alreadyConsumed,
|
||||
void* dst, size_t dstCapacity)
|
||||
{
|
||||
int firstChunk = 1;
|
||||
size_t ret = 1;
|
||||
|
||||
assert(f_in != NULL); assert(f_out != NULL);
|
||||
assert(dctx != NULL);
|
||||
assert(src != NULL); assert(srcCapacity > 0); assert(filled <= srcCapacity); assert(alreadyConsumed <= filled);
|
||||
assert(dst != NULL); assert(dstCapacity > 0);
|
||||
|
||||
/* Decompression */
|
||||
while (ret != 0) {
|
||||
/* Load more input */
|
||||
size_t readSize = firstChunk ? filled : fread(src, 1, srcCapacity, f_in); firstChunk=0;
|
||||
const void* srcPtr = (const char*)src + alreadyConsumed; alreadyConsumed=0;
|
||||
const void* const srcEnd = (const char*)srcPtr + readSize;
|
||||
if (readSize == 0 || ferror(f_in)) {
|
||||
printf("Decompress: not enough input or error reading file\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
/* Decompress:
|
||||
* Continue while there is more input to read (srcPtr != srcEnd)
|
||||
* and the frame isn't over (ret != 0)
|
||||
*/
|
||||
while (srcPtr < srcEnd && ret != 0) {
|
||||
/* Any data within dst has been flushed at this stage */
|
||||
size_t dstSize = dstCapacity;
|
||||
size_t srcSize = (const char*)srcEnd - (const char*)srcPtr;
|
||||
ret = LZ4F_decompress(dctx, dst, &dstSize, srcPtr, &srcSize, /* LZ4F_decompressOptions_t */ NULL);
|
||||
if (LZ4F_isError(ret)) {
|
||||
printf("Decompression error: %s\n", LZ4F_getErrorName(ret));
|
||||
return 1;
|
||||
}
|
||||
/* Flush output */
|
||||
if (dstSize != 0) safe_fwrite(dst, 1, dstSize, f_out);
|
||||
/* Update input */
|
||||
srcPtr = (const char*)srcPtr + srcSize;
|
||||
}
|
||||
|
||||
assert(srcPtr <= srcEnd);
|
||||
|
||||
/* Ensure all input data has been consumed.
|
||||
* It is valid to have multiple frames in the same file,
|
||||
* but this example only supports one frame.
|
||||
*/
|
||||
if (srcPtr < srcEnd) {
|
||||
printf("Decompress: Trailing data left in file after frame\n");
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
|
||||
/* Check that there isn't trailing data in the file after the frame.
|
||||
* It is valid to have multiple frames in the same file,
|
||||
* but this example only supports one frame.
|
||||
*/
|
||||
{ size_t const readSize = fread(src, 1, 1, f_in);
|
||||
if (readSize != 0 || !feof(f_in)) {
|
||||
printf("Decompress: Trailing data left in file after frame\n");
|
||||
return 1;
|
||||
} }
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
/* @return : 1==error, 0==completed */
|
||||
static int
|
||||
decompress_file_allocDst(FILE* f_in, FILE* f_out,
|
||||
LZ4F_dctx* dctx,
|
||||
void* src, size_t srcCapacity)
|
||||
{
|
||||
assert(f_in != NULL); assert(f_out != NULL);
|
||||
assert(dctx != NULL);
|
||||
assert(src != NULL);
|
||||
assert(srcCapacity >= LZ4F_HEADER_SIZE_MAX); /* ensure LZ4F_getFrameInfo() can read enough data */
|
||||
|
||||
/* Read Frame header */
|
||||
size_t const readSize = fread(src, 1, srcCapacity, f_in);
|
||||
if (readSize == 0 || ferror(f_in)) {
|
||||
printf("Decompress: not enough input or error reading file\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
LZ4F_frameInfo_t info;
|
||||
size_t consumedSize = readSize;
|
||||
{ size_t const fires = LZ4F_getFrameInfo(dctx, &info, src, &consumedSize);
|
||||
if (LZ4F_isError(fires)) {
|
||||
printf("LZ4F_getFrameInfo error: %s\n", LZ4F_getErrorName(fires));
|
||||
return 1;
|
||||
} }
|
||||
|
||||
/* Allocating enough space for an entire block isn't necessary for
|
||||
* correctness, but it allows some memcpy's to be elided.
|
||||
*/
|
||||
size_t const dstCapacity = get_block_size(&info);
|
||||
void* const dst = malloc(dstCapacity);
|
||||
if (!dst) { perror("decompress_file(dst)"); return 1; }
|
||||
|
||||
int const decompressionResult = decompress_file_internal(
|
||||
f_in, f_out,
|
||||
dctx,
|
||||
src, srcCapacity, readSize-consumedSize, consumedSize,
|
||||
dst, dstCapacity);
|
||||
|
||||
free(dst);
|
||||
return decompressionResult;
|
||||
}
|
||||
|
||||
|
||||
/* @result : 1==error, 0==success */
|
||||
static int decompress_file(FILE* f_in, FILE* f_out)
|
||||
{
|
||||
assert(f_in != NULL); assert(f_out != NULL);
|
||||
|
||||
/* Resource allocation */
|
||||
void* const src = malloc(IN_CHUNK_SIZE);
|
||||
if (!src) { perror("decompress_file(src)"); return 1; }
|
||||
|
||||
LZ4F_dctx* dctx;
|
||||
{ size_t const dctxStatus = LZ4F_createDecompressionContext(&dctx, LZ4F_VERSION);
|
||||
if (LZ4F_isError(dctxStatus)) {
|
||||
printf("LZ4F_dctx creation error: %s\n", LZ4F_getErrorName(dctxStatus));
|
||||
} }
|
||||
|
||||
int const result = !dctx ? 1 /* error */ :
|
||||
decompress_file_allocDst(f_in, f_out, dctx, src, IN_CHUNK_SIZE);
|
||||
|
||||
free(src);
|
||||
LZ4F_freeDecompressionContext(dctx); /* note : free works on NULL */
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
int compareFiles(FILE* fp0, FILE* fp1, FILE* fpUnc, long uncOffset)
|
||||
{
|
||||
int result = 0;
|
||||
long bytesRead = 0;
|
||||
long bytesToOffset = -1;
|
||||
long b1Size = 1024;
|
||||
|
||||
while (result==0) {
|
||||
char b1[b1Size];
|
||||
size_t r1;
|
||||
size_t bytesToRead = sizeof b1;
|
||||
if (uncOffset >= 0) {
|
||||
bytesToOffset = uncOffset - bytesRead;
|
||||
|
||||
/* read remainder to offset */
|
||||
if (bytesToOffset < b1Size) {
|
||||
bytesToRead = bytesToOffset;
|
||||
}
|
||||
}
|
||||
|
||||
char b0[1024];
|
||||
size_t r0;
|
||||
if (bytesToOffset <= 0 && fpUnc) {
|
||||
bytesToRead = sizeof b1;
|
||||
r0 = fread(b0, 1,bytesToRead, fpUnc);
|
||||
} else {
|
||||
r0 = fread(b0, 1, bytesToRead, fp0);
|
||||
}
|
||||
|
||||
r1 = fread(b1, 1, r0, fp1);
|
||||
|
||||
result = (r0 != r1);
|
||||
if (!r0 || !r1) break;
|
||||
if (!result) result = memcmp(b0, b1, r0);
|
||||
|
||||
bytesRead += r1;
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
char inpFilename[256] = { 0 };
|
||||
char lz4Filename[256] = { 0 };
|
||||
char decFilename[256] = { 0 };
|
||||
|
||||
int uncOffset = -1;
|
||||
char uncFilename[256] = { 0 };
|
||||
int opt;
|
||||
|
||||
if (argc < 2) {
|
||||
printf("Please specify input filename\n");
|
||||
return EXIT_FAILURE;
|
||||
}
|
||||
|
||||
snprintf(inpFilename, 256, "%s", argv[1]);
|
||||
snprintf(lz4Filename, 256, "%s.lz4", argv[1]);
|
||||
snprintf(decFilename, 256, "%s.lz4.dec", argv[1]);
|
||||
|
||||
while ((opt = getopt(argc, argv, "o:d:")) != -1) {
|
||||
switch (opt) {
|
||||
case 'd':
|
||||
snprintf(uncFilename, 256, "%s", optarg);
|
||||
break;
|
||||
case 'o':
|
||||
uncOffset = atoi(optarg);
|
||||
break;
|
||||
default:
|
||||
printf("usage: %s <input file> [-o <offset> -d <file>]\n", argv[0]);
|
||||
printf("-o uncompressed data offset\n");
|
||||
printf(" inject uncompressed data at this offset into the lz4 file\n");
|
||||
printf("-d uncompressed file\n");
|
||||
printf(" file to inject without compression into the lz4 file\n");
|
||||
return EXIT_FAILURE;
|
||||
}
|
||||
}
|
||||
|
||||
printf("inp = [%s]\n", inpFilename);
|
||||
printf("lz4 = [%s]\n", lz4Filename);
|
||||
printf("dec = [%s]\n", decFilename);
|
||||
if (uncOffset > 0) {
|
||||
printf("unc = [%s]\n", uncFilename);
|
||||
printf("ofs = [%i]\n", uncOffset);
|
||||
}
|
||||
|
||||
/* compress */
|
||||
{ FILE* const inpFp = fopen(inpFilename, "rb");
|
||||
FILE* const outFp = fopen(lz4Filename, "wb");
|
||||
FILE* const uncFp = fopen(uncFilename, "rb");
|
||||
|
||||
printf("compress : %s -> %s\n", inpFilename, lz4Filename);
|
||||
compressResult_t const ret = compress_file(
|
||||
inpFp, outFp,
|
||||
uncFp, uncOffset);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
if (uncFp)
|
||||
fclose(uncFp);
|
||||
|
||||
if (ret.error) {
|
||||
printf("compress : failed with code %i\n", ret.error);
|
||||
return ret.error;
|
||||
}
|
||||
printf("%s: %zu → %zu bytes, %.1f%%\n",
|
||||
inpFilename,
|
||||
(size_t)ret.size_in, (size_t)ret.size_out, /* might overflow is size_t is 32 bits and size_{in,out} > 4 GB */
|
||||
(double)ret.size_out / ret.size_in * 100);
|
||||
printf("compress : done\n");
|
||||
}
|
||||
|
||||
/* decompress */
|
||||
{ FILE* const inpFp = fopen(lz4Filename, "rb");
|
||||
FILE* const outFp = fopen(decFilename, "wb");
|
||||
|
||||
printf("decompress : %s -> %s\n", lz4Filename, decFilename);
|
||||
int const ret = decompress_file(inpFp, outFp);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
|
||||
if (ret) {
|
||||
printf("decompress : failed with code %i\n", ret);
|
||||
return ret;
|
||||
}
|
||||
printf("decompress : done\n");
|
||||
}
|
||||
|
||||
/* verify */
|
||||
{ FILE* const inpFp = fopen(inpFilename, "rb");
|
||||
FILE* const decFp = fopen(decFilename, "rb");
|
||||
FILE* const uncFp = fopen(uncFilename, "rb");
|
||||
|
||||
printf("verify : %s <-> %s\n", inpFilename, decFilename);
|
||||
int const cmp = compareFiles(inpFp, decFp,
|
||||
uncFp, uncOffset);
|
||||
|
||||
fclose(decFp);
|
||||
fclose(inpFp);
|
||||
if (uncFp)
|
||||
fclose(uncFp);
|
||||
|
||||
if (cmp) {
|
||||
printf("corruption detected : decompressed file differs from original\n");
|
||||
return cmp;
|
||||
}
|
||||
printf("verify : OK\n");
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
13
expkg/vendor/lz4/examples/print_version.c
vendored
Normal file
13
expkg/vendor/lz4/examples/print_version.c
vendored
Normal file
@ -0,0 +1,13 @@
|
||||
// LZ4 trivial example : print Library version number
|
||||
// by Takayuki Matsuoka
|
||||
|
||||
|
||||
#include <stdio.h>
|
||||
#include "lz4.h"
|
||||
|
||||
int main(int argc, char** argv)
|
||||
{
|
||||
(void)argc; (void)argv;
|
||||
printf("Hello World ! LZ4 Library version = %d\n", LZ4_versionNumber());
|
||||
return 0;
|
||||
}
|
||||
99
expkg/vendor/lz4/examples/simple_buffer.c
vendored
Normal file
99
expkg/vendor/lz4/examples/simple_buffer.c
vendored
Normal file
@ -0,0 +1,99 @@
|
||||
/*
|
||||
* simple_buffer.c
|
||||
* Copyright : Kyle Harper
|
||||
* License : Follows same licensing as the lz4.c/lz4.h program at any given time. Currently, BSD 2.
|
||||
* Description: Example program to demonstrate the basic usage of the compress/decompress functions within lz4.c/lz4.h.
|
||||
* The functions you'll likely want are LZ4_compress_default and LZ4_decompress_safe.
|
||||
* Both of these are documented in the lz4.h header file; I recommend reading them.
|
||||
*/
|
||||
|
||||
/* Dependencies */
|
||||
#include <stdio.h> // For printf()
|
||||
#include <string.h> // For memcmp()
|
||||
#include <stdlib.h> // For exit()
|
||||
#include "lz4.h" // This is all that is required to expose the prototypes for basic compression and decompression.
|
||||
|
||||
/*
|
||||
* Simple show-error-and-bail function.
|
||||
*/
|
||||
void run_screaming(const char* message, const int code) {
|
||||
printf("%s \n", message);
|
||||
exit(code);
|
||||
}
|
||||
|
||||
|
||||
/*
|
||||
* main
|
||||
*/
|
||||
int main(void) {
|
||||
/* Introduction */
|
||||
// Below we will have a Compression and Decompression section to demonstrate.
|
||||
// There are a few important notes before we start:
|
||||
// 1) The return codes of LZ4_ functions are important.
|
||||
// Read lz4.h if you're unsure what a given code means.
|
||||
// 2) LZ4 uses char* pointers in all LZ4_ functions.
|
||||
// This is baked into the API and not going to change, for consistency.
|
||||
// If your program uses different pointer types,
|
||||
// you may need to do some casting or set the right -Wno compiler flags to ignore those warnings (e.g.: -Wno-pointer-sign).
|
||||
|
||||
/* Compression */
|
||||
// We'll store some text into a variable pointed to by *src to be compressed later.
|
||||
const char* const src = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Lorem ipsum dolor site amat.";
|
||||
// The compression function needs to know how many bytes exist. Since we're using a string, we can use strlen() + 1 (for \0).
|
||||
const int src_size = (int)(strlen(src) + 1);
|
||||
// LZ4 provides a function that will tell you the maximum size of compressed output based on input data via LZ4_compressBound().
|
||||
const int max_dst_size = LZ4_compressBound(src_size);
|
||||
// We will use that size for our destination boundary when allocating space.
|
||||
char* compressed_data = (char*)malloc((size_t)max_dst_size);
|
||||
if (compressed_data == NULL)
|
||||
run_screaming("Failed to allocate memory for *compressed_data.", 1);
|
||||
// That's all the information and preparation LZ4 needs to compress *src into* compressed_data.
|
||||
// Invoke LZ4_compress_default now with our size values and pointers to our memory locations.
|
||||
// Save the return value for error checking.
|
||||
const int compressed_data_size = LZ4_compress_default(src, compressed_data, src_size, max_dst_size);
|
||||
// Check return_value to determine what happened.
|
||||
if (compressed_data_size <= 0)
|
||||
run_screaming("A 0 or negative result from LZ4_compress_default() indicates a failure trying to compress the data. ", 1);
|
||||
if (compressed_data_size > 0)
|
||||
printf("We successfully compressed some data! Ratio: %.2f\n",
|
||||
(float) compressed_data_size/src_size);
|
||||
// Not only does a positive return_value mean success, the value returned == the number of bytes required.
|
||||
// You can use this to realloc() *compress_data to free up memory, if desired. We'll do so just to demonstrate the concept.
|
||||
compressed_data = (char *)realloc(compressed_data, (size_t)compressed_data_size);
|
||||
if (compressed_data == NULL)
|
||||
run_screaming("Failed to re-alloc memory for compressed_data. Sad :(", 1);
|
||||
|
||||
|
||||
/* Decompression */
|
||||
// Now that we've successfully compressed the information from *src to *compressed_data, let's do the opposite!
|
||||
// The decompression will need to know the compressed size, and an upper bound of the decompressed size.
|
||||
// In this example, we just re-use this information from previous section,
|
||||
// but in a real-world scenario, metadata must be transmitted to the decompression side.
|
||||
// Each implementation is in charge of this part. Oftentimes, it adds some header of its own.
|
||||
// Sometimes, the metadata can be extracted from the local context.
|
||||
|
||||
// First, let's create a *new_src location of size src_size since we know that value.
|
||||
char* const regen_buffer = (char*)malloc(src_size);
|
||||
if (regen_buffer == NULL)
|
||||
run_screaming("Failed to allocate memory for *regen_buffer.", 1);
|
||||
// The LZ4_decompress_safe function needs to know where the compressed data is, how many bytes long it is,
|
||||
// where the regen_buffer memory location is, and how large regen_buffer (uncompressed) output will be.
|
||||
// Again, save the return_value.
|
||||
const int decompressed_size = LZ4_decompress_safe(compressed_data, regen_buffer, compressed_data_size, src_size);
|
||||
free(compressed_data); /* no longer useful */
|
||||
if (decompressed_size < 0)
|
||||
run_screaming("A negative result from LZ4_decompress_safe indicates a failure trying to decompress the data. See exit code (echo $?) for value returned.", decompressed_size);
|
||||
if (decompressed_size >= 0)
|
||||
printf("We successfully decompressed some data!\n");
|
||||
// Not only does a positive return value mean success,
|
||||
// value returned == number of bytes regenerated from compressed_data stream.
|
||||
if (decompressed_size != src_size)
|
||||
run_screaming("Decompressed data is different from original! \n", 1);
|
||||
|
||||
/* Validation */
|
||||
// We should be able to compare our original *src with our *new_src and be byte-for-byte identical.
|
||||
if (memcmp(src, regen_buffer, src_size) != 0)
|
||||
run_screaming("Validation failed. *src and *new_src are not identical.", 1);
|
||||
printf("Validation done. The string we ended up with is:\n%s\n", regen_buffer);
|
||||
return 0;
|
||||
}
|
||||
238
expkg/vendor/lz4/examples/streamingHC_ringBuffer.c
vendored
Normal file
238
expkg/vendor/lz4/examples/streamingHC_ringBuffer.c
vendored
Normal file
@ -0,0 +1,238 @@
|
||||
// LZ4 HC streaming API example : ring buffer
|
||||
// Based on a previous example by Takayuki Matsuoka
|
||||
|
||||
|
||||
/**************************************
|
||||
* Compiler Options
|
||||
**************************************/
|
||||
#if defined(_MSC_VER) && (_MSC_VER <= 1800) /* Visual Studio <= 2013 */
|
||||
# define _CRT_SECURE_NO_WARNINGS
|
||||
# define snprintf sprintf_s
|
||||
#endif
|
||||
|
||||
#define GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__)
|
||||
#ifdef __GNUC__
|
||||
# pragma GCC diagnostic ignored "-Wmissing-braces" /* GCC bug 53119 : doesn't accept { 0 } as initializer (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=53119) */
|
||||
#endif
|
||||
|
||||
|
||||
/**************************************
|
||||
* Includes
|
||||
**************************************/
|
||||
#include "lz4hc.h"
|
||||
#include "lz4.h"
|
||||
|
||||
#include <stdio.h>
|
||||
#include <stdint.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <assert.h>
|
||||
|
||||
enum {
|
||||
MESSAGE_MAX_BYTES = 1024,
|
||||
RING_BUFFER_BYTES = 1024 * 8 + MESSAGE_MAX_BYTES,
|
||||
DEC_BUFFER_BYTES = RING_BUFFER_BYTES + MESSAGE_MAX_BYTES // Intentionally larger to test unsynchronized ring buffers
|
||||
};
|
||||
|
||||
|
||||
size_t write_int32(FILE* fp, int32_t i) {
|
||||
return fwrite(&i, sizeof(i), 1, fp);
|
||||
}
|
||||
|
||||
size_t write_bin(FILE* fp, const void* array, int arrayBytes) {
|
||||
assert(arrayBytes >= 0);
|
||||
return fwrite(array, 1, (size_t)arrayBytes, fp);
|
||||
}
|
||||
|
||||
size_t read_int32(FILE* fp, int32_t* i) {
|
||||
return fread(i, sizeof(*i), 1, fp);
|
||||
}
|
||||
|
||||
size_t read_bin(FILE* fp, void* array, int arrayBytes) {
|
||||
assert(arrayBytes >= 0);
|
||||
return fread(array, 1, (size_t)arrayBytes, fp);
|
||||
}
|
||||
|
||||
|
||||
void test_compress(FILE* outFp, FILE* inpFp)
|
||||
{
|
||||
assert(outFp != NULL); assert(inpFp != NULL);
|
||||
|
||||
LZ4_streamHC_t lz4Stream_body = { 0 };
|
||||
LZ4_streamHC_t* lz4Stream = &lz4Stream_body;
|
||||
|
||||
static char inpBuf[RING_BUFFER_BYTES];
|
||||
int inpOffset = 0;
|
||||
|
||||
for(;;) {
|
||||
// Read random length ([1,MESSAGE_MAX_BYTES]) data to the ring buffer.
|
||||
char* const inpPtr = &inpBuf[inpOffset];
|
||||
const int randomLength = (rand() % MESSAGE_MAX_BYTES) + 1;
|
||||
const int inpBytes = (int) read_bin(inpFp, inpPtr, randomLength);
|
||||
if (0 == inpBytes) break;
|
||||
|
||||
#define CMPBUFSIZE (LZ4_COMPRESSBOUND(MESSAGE_MAX_BYTES))
|
||||
{ char cmpBuf[CMPBUFSIZE];
|
||||
const int cmpBytes = LZ4_compress_HC_continue(lz4Stream, inpPtr, cmpBuf, inpBytes, CMPBUFSIZE);
|
||||
|
||||
if(cmpBytes <= 0) break;
|
||||
write_int32(outFp, cmpBytes);
|
||||
write_bin(outFp, cmpBuf, cmpBytes);
|
||||
|
||||
inpOffset += inpBytes;
|
||||
|
||||
// Wraparound the ringbuffer offset
|
||||
if(inpOffset >= RING_BUFFER_BYTES - MESSAGE_MAX_BYTES)
|
||||
inpOffset = 0;
|
||||
}
|
||||
}
|
||||
|
||||
write_int32(outFp, 0);
|
||||
}
|
||||
|
||||
|
||||
void test_decompress(FILE* outFp, FILE* inpFp)
|
||||
{
|
||||
assert(outFp != NULL); assert(inpFp != NULL);
|
||||
|
||||
static char decBuf[DEC_BUFFER_BYTES];
|
||||
int decOffset = 0;
|
||||
LZ4_streamDecode_t lz4StreamDecode_body = { 0 };
|
||||
LZ4_streamDecode_t* lz4StreamDecode = &lz4StreamDecode_body;
|
||||
|
||||
for(;;) {
|
||||
int cmpBytes = 0;
|
||||
char cmpBuf[CMPBUFSIZE];
|
||||
|
||||
{ const size_t r0 = read_int32(inpFp, &cmpBytes);
|
||||
size_t r1;
|
||||
if(r0 != 1 || cmpBytes <= 0)
|
||||
break;
|
||||
|
||||
r1 = read_bin(inpFp, cmpBuf, cmpBytes);
|
||||
if(r1 != (size_t) cmpBytes)
|
||||
break;
|
||||
}
|
||||
|
||||
{ char* const decPtr = &decBuf[decOffset];
|
||||
const int decBytes = LZ4_decompress_safe_continue(
|
||||
lz4StreamDecode, cmpBuf, decPtr, cmpBytes, MESSAGE_MAX_BYTES);
|
||||
if(decBytes <= 0)
|
||||
break;
|
||||
|
||||
decOffset += decBytes;
|
||||
write_bin(outFp, decPtr, decBytes);
|
||||
|
||||
// Wraparound the ringbuffer offset
|
||||
if(decOffset >= DEC_BUFFER_BYTES - MESSAGE_MAX_BYTES)
|
||||
decOffset = 0;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
// Compare 2 files content
|
||||
// return 0 if identical
|
||||
// return ByteNb>0 if different
|
||||
size_t compare(FILE* f0, FILE* f1)
|
||||
{
|
||||
assert(f0 != NULL); assert(f1 != NULL);
|
||||
|
||||
size_t result = 1;
|
||||
|
||||
for (;;) {
|
||||
char b0[65536];
|
||||
char b1[65536];
|
||||
const size_t r0 = fread(b0, 1, sizeof(b0), f0);
|
||||
const size_t r1 = fread(b1, 1, sizeof(b1), f1);
|
||||
|
||||
if ((r0==0) && (r1==0)) return 0; // success
|
||||
|
||||
if (r0 != r1) {
|
||||
size_t smallest = r0;
|
||||
if (r1<r0) smallest = r1;
|
||||
result += smallest;
|
||||
break;
|
||||
}
|
||||
|
||||
if (memcmp(b0, b1, r0)) {
|
||||
unsigned errorPos = 0;
|
||||
while ((errorPos < r0) && (b0[errorPos]==b1[errorPos])) errorPos++;
|
||||
result += errorPos;
|
||||
break;
|
||||
}
|
||||
|
||||
result += sizeof(b0);
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
|
||||
int main(int argc, const char** argv)
|
||||
{
|
||||
char inpFilename[256] = { 0 };
|
||||
char lz4Filename[256] = { 0 };
|
||||
char decFilename[256] = { 0 };
|
||||
unsigned fileID = 1;
|
||||
unsigned pause = 0;
|
||||
|
||||
|
||||
if(argc < 2) {
|
||||
printf("Please specify input filename\n");
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!strcmp(argv[1], "-p")) { pause = 1; fileID = 2; }
|
||||
|
||||
snprintf(inpFilename, 256, "%s", argv[fileID]);
|
||||
snprintf(lz4Filename, 256, "%s.lz4s-%d", argv[fileID], 9);
|
||||
snprintf(decFilename, 256, "%s.lz4s-%d.dec", argv[fileID], 9);
|
||||
|
||||
printf("input = [%s]\n", inpFilename);
|
||||
printf("lz4 = [%s]\n", lz4Filename);
|
||||
printf("decoded = [%s]\n", decFilename);
|
||||
|
||||
// compress
|
||||
{ FILE* const inpFp = fopen(inpFilename, "rb");
|
||||
FILE* const outFp = fopen(lz4Filename, "wb");
|
||||
|
||||
test_compress(outFp, inpFp);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
// decompress
|
||||
{ FILE* const inpFp = fopen(lz4Filename, "rb");
|
||||
FILE* const outFp = fopen(decFilename, "wb");
|
||||
|
||||
test_decompress(outFp, inpFp);
|
||||
|
||||
fclose(outFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
// verify
|
||||
{ FILE* const inpFp = fopen(inpFilename, "rb");
|
||||
FILE* const decFp = fopen(decFilename, "rb");
|
||||
|
||||
const size_t cmp = compare(inpFp, decFp);
|
||||
if(0 == cmp) {
|
||||
printf("Verify : OK\n");
|
||||
} else {
|
||||
printf("Verify : NG : error at pos %u\n", (unsigned)cmp-1);
|
||||
}
|
||||
|
||||
fclose(decFp);
|
||||
fclose(inpFp);
|
||||
}
|
||||
|
||||
if (pause) {
|
||||
int unused;
|
||||
printf("Press enter to continue ...\n");
|
||||
unused = getchar(); (void)unused; /* silence static analyzer */
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
87
expkg/vendor/lz4/examples/streaming_api_basics.md
vendored
Normal file
87
expkg/vendor/lz4/examples/streaming_api_basics.md
vendored
Normal file
@ -0,0 +1,87 @@
|
||||
# LZ4 Streaming API Basics
|
||||
by *Takayuki Matsuoka*
|
||||
## LZ4 API sets
|
||||
|
||||
LZ4 has the following API sets :
|
||||
|
||||
- "Auto Framing" API (lz4frame.h) :
|
||||
This is most recommended API for usual application.
|
||||
It guarantees interoperability with other LZ4 framing format compliant tools/libraries
|
||||
such as LZ4 command line utility, node-lz4, etc.
|
||||
- "Block" API : This is recommended for simple purpose.
|
||||
It compresses single raw memory block to LZ4 memory block and vice versa.
|
||||
- "Streaming" API : This is designed for complex things.
|
||||
For example, compress huge stream data in restricted memory environment.
|
||||
|
||||
Basically, you should use "Auto Framing" API.
|
||||
But if you want to write advanced application, it's time to use Block or Streaming APIs.
|
||||
|
||||
|
||||
## What is difference between Block and Streaming API ?
|
||||
|
||||
Block API (de)compresses a single contiguous memory block.
|
||||
In other words, LZ4 library finds redundancy from a single contiguous memory block.
|
||||
Streaming API does same thing but (de)compresses multiple adjacent contiguous memory blocks.
|
||||
So Streaming API could find more redundancy than Block API.
|
||||
|
||||
The following figure shows difference between API and block sizes.
|
||||
In these figures, the original data is split into 4KiBytes contiguous chunks.
|
||||
|
||||
```
|
||||
Original Data
|
||||
+---------------+---------------+----+----+----+
|
||||
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
|
||||
+---------------+---------------+----+----+----+
|
||||
|
||||
Example (1) : Block API, 4KiB Block
|
||||
+---------------+---------------+----+----+----+
|
||||
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
|
||||
+---------------+---------------+----+----+----+
|
||||
| Block #1 | Block #2 | #3 | #4 |... |
|
||||
+---------------+---------------+----+----+----+
|
||||
|
||||
(No Dependency)
|
||||
|
||||
|
||||
Example (2) : Block API, 8KiB Block
|
||||
+---------------+---------------+----+----+----+
|
||||
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
|
||||
+---------------+---------------+----+----+----+
|
||||
| Block #1 |Block #2 |... |
|
||||
+--------------------+----------+-------+-+----+
|
||||
^ | ^ |
|
||||
| | | |
|
||||
+--------------+ +----+
|
||||
Internal Dependency Internal Dependency
|
||||
|
||||
|
||||
Example (3) : Streaming API, 4KiB Block
|
||||
+---------------+---------------+-----+----+----+
|
||||
| 4KiB Chunk A | 4KiB Chunk B | C | D |... |
|
||||
+---------------+---------------+-----+----+----+
|
||||
| Block #1 | Block #2 | #3 | #4 |... |
|
||||
+---------------+----+----------+-+---+-+--+----+
|
||||
^ | ^ | ^ |
|
||||
| | | | | |
|
||||
+--------------+ +--------+ +---+
|
||||
Dependency Dependency Dependency
|
||||
```
|
||||
|
||||
- In example (1), there is no dependency.
|
||||
All blocks are compressed independently.
|
||||
- In example (2), naturally 8KiBytes block has internal dependency.
|
||||
But still block #1 and #2 are compressed independently.
|
||||
- In example (3), block #2 has dependency to #1,
|
||||
also #3 has dependency to #2 and #1, #4 has #3, #2 and #1, and so on.
|
||||
|
||||
Here, we can observe difference between example (2) and (3).
|
||||
In (2), there's no dependency between chunk B and C, but (3) has dependency between B and C.
|
||||
This dependency improves compression ratio.
|
||||
|
||||
|
||||
## Restriction of Streaming API
|
||||
|
||||
For efficiency, Streaming API doesn't keep a mirror copy of dependent (de)compressed memory.
|
||||
This means users should keep these dependent (de)compressed memory explicitly.
|
||||
Usually, "Dependent memory" is previous adjacent contiguous memory up to 64KiBytes.
|
||||
LZ4 will not access further memories.
|
||||
Reference in New Issue
Block a user