libpqcrypto: Changes

Version 20180314

This is the beginning of the libpqcrypto changelog. Normally the beginning of a changelog is empty, but libpqcrypto started with software submitted to NIST; what follows is a log of changes made from that software to the corresponding components of libpqcrypto.

Namespacing

Each global symbol defined in libpqcrypto, and each header file provided by libpqcrypto, is in one of the following namespaces: pqcrypto; pqrandombytes; pqkernelrandombytes. The libpqcrypto compilation script issues warnings for any violations of this rule.

Implementations now follow this rule on several tested platforms (but this has not yet been comprehensively enforced at the source-code level and might still fail on other platforms). For externally visible functions, .c files now include crypto_kem.h, crypto_sign.h, randombytes.h, etc. For internal functions, new namespacing files use #define to move each internal function name into a private namespace, and are used via -imacros namespacing. Various .s files are now .S and also use the namespacing macros.

Some asm names were manually assigned in avx2 implementations in dilithium*, kyber*, newhope*, and ntruhrss*. These assignments are now gone in favor of namespacing.

Global symbols outside the defined API now have hidden visibility, preventing interposition when libpqcrypto is used as a shared library.

Randomness and tests

Various RNG software layers are now gone, including randombytes.c and randombytes.h in dilithium*; random.c in frodo*; prng_seed* in gui* and rainbow*; rand_bytes and randomness.h in picnic*; and various copies of NIST's rng.c and rng.h. All randomness is now obtained from randombytes(), provided by #include "randombytes.h". libpqcrypto includes a centralized randombytes() implementation, the same as fastrandombytes from SUPERCOP; and a centralized deterministic randombytes() implementation for checksums, the same as knownrandombytes from SUPERCOP. The deterministic implementation is used only for tests and is not included in -lpqcrypto.

The NIST KAT-generation code (PQCgen* and the simplified kat_*) is now gone. Other test drivers (test.c, test_qtesla.c, PQCtestKAT_sign.c, etc.) are also gone. The library has a centralized test/checksum mechanism, computed the same way as in SUPERCOP.

Compilation instructions

Each per-implementation Makefile is now gone. These files were used for several purposes:

Specifying compiler choices (e.g., gcc -Ofast) and prerequisite libraries (e.g., -lkeccak -lcrypto). libpqcrypto handles this centrally.
Specifying files to compile. libpqcrypto, like NaCl and SUPERCOP, always compiles all .c, .s, and .S files in the top directory of an implementation. Files in the aes and sha3 subdirectories for frodo*, and the sha3 subdirectory for qtesla*, are now in the top directory.
Specifying files to not compile. libpqcrypto, like NaCl and SUPERCOP, does not compile files outside the top directory. Included files in the top directory under the name *.c that were not meant to be compiled directly (e.g., poly_mul.c in saber) are now renamed *.inc.
For frodo*: Specifying various macros. Some of these were unused and are now eliminated. Others are now incorporated into .h files.

Bug fixes and portability improvements

dags* had a printf for the occasional "Non systematic matrix". This was caught by the automatic tests and is now removed.

frodo*/x64 had some vectorized _load_ and _store_ (rather than the safe _loadu_ and _storeu_) on data that was not necessarily aligned, crashing when the data was not aligned. This was caught by the automatic tests. The relevant arrays are now aligned.

gui* had a stack buffer overflow. This was caught by Address Sanitizer and is now fixed.

The luov*/portable implementation leaked memory. This was caught by valgrind and is now fixed.

mceliece*/avx used 0X for quad hex values in consts.S, and now uses 0x. 0X works with gcc but not clang.

mqdss64 was reading uninitialized data. This was caught by valgrind (and also indirectly by other tests) and is now fixed.

qtesla* allocated mlen bytes on the stack, crashing for messages above about 4 megabytes. This is now handled with malloc.

rainbow*a had a stack buffer overflow. This was caught by Address Sanitizer and is now fixed.

ramstake756839 wrote a zero byte past the end of the secret-key buffer. This was caught by the automatic tests and is now fixed.

sphincs* now includes various post-submission code updates.

Following existing interface rules

kindi* included a crypto_encrypt interface, but ignored the message length provided as input in that API, and instead assumed a fixed-length message. This was caught by the automatic tests. libpqcrypto provides only crypto_kem and crypto_sign, not crypto_encrypt.

dilithium*, gui*, luov*, mqdss*, qtesla*, rainbow*, and sphincs* did not allow the message pointer to match the signed-message pointer. This was caught by the automatic tests in some cases, including all cases where signatures were shorter than the message lengths in the tests. memcpy is now replaced by memmove where appropriate, and in some cases crypto_sign_open now copies the incoming signature to a temporary buffer.

dilithium*, gui*, and rainbow* did not allow the public-key pointer to match the output pointer in crypto_sign_open. This was caught by the automatic tests. crypto_sign_open now copies the public key to a temporary buffer.

Following additional interface rules

api.h is now stripped down to numeric definitions of CRYPTO_BYTES etc., so it can be easily parsed without C preprocessing.

The NIST submission rules were less restrictive and allowed api.h to be used as a general-purpose configuration mechanism. Various .c files that included api.h now include apiorig.h instead, with the original api.h renamed as apiorig.h. Probably some of these apiorig.h files can be removed, but this cleanup has not happened yet.

Each kem primitive now has a goal-indcca2 file meaning that it tries to provide IND-CCA2 security (libpqcrypto does not include newhope*cpa), and a goal-indcpa file meaning that it tries to provide IND-CPA security (which is implied by IND-CCA2 security). However, the quantitative target security level is not indicated.

mceliece*/avx, newhope*/avx2, and ntruhrss*/avx2 now use rip-relative addressing for constants in memory used in assembly. Previously they used absolute addressing, which works in a static library but not in a shared library.

Precomputed constants in dilithium*, gui*, kyber*, luov*, mceliece*, newhope*, ntruhrss*, qtesla*, rainbow*, and ramstake* are now defined as const (equivalently, .section .rodata in assembly) so that they are placed in the text segment and shared across processes. This is not comprehensive: some implementations of some primitives have variables in the data segment, and some functions (notably randombytes) are not thread-safe.

Fewer compiler warnings

By default, libpqcrypto compiles with -Wall with both gcc and clang. The following changes reduce the volume of warnings:

gui*: The unused num_nonzero_terms function is now removed.
luov* now closes comments in parameters.h. Various unused variables are now removed. An initializer {0} is replaced with {{0}} (and could simply be omitted).
qtesla*: Various unused variables are now removed.
sphincs* now has a revised TRUNCSTORE definition.
picnic* now says #ifndef api_h instead of #ifndef api_.

More centralization

Most extracts from the Keccak Code Package (e.g., in picnic*) are now gone. The library has a centralized copy of a larger extract from the Keccak Code Package. However:

KeccakP-1600-times4-SIMD256.c is still included in individual implementations since the Keccak Code Package does not have a portable implementation of the underlying KeccakP1600times4_PermuteAll_24rounds function.
The Keccak Code Package is only one of the SHA-3 implementations; these implementations have not yet been merged.

cpucycles() implementations are now gone. The library has a centralized (and more portable) cpucycles().

Version: This is version 2018.03.14 of the "Changes" web page.