• Version: 6.4.0 - 8.0.0-pre
• Platform:
• Subsystem:

Several hash functions hard crash when supplied 'ucs2' encodings.

Snippet for Hash:

    crypto.createHash('sha256').digest('ucs2');

Snippet for Hmac:

    crypto.createHmac('sha256', 'w00t').digest('ucs2');

This is because for both the binding layer functions end up calling
StringBytes::Encode with UCS2, which has a hard check:

  CHECK_NE(encoding, UCS2); // <- this can be controlled from JS
  CHECK_LE(buflen, Buffer::kMaxLength);

The Sign::SignFinal binding function does this too, but the js wrapper always
calls it with the encoding set to null. So you'd have to call handle directly
to crash:

    const private_key = '-----BEGIN EC PRIVATE KEY-----\n' +
      'MHcCAQEEIF+jnWY1D5kbVYDNvxxo/Y+ku2uJPDwS0r/VuPZQrjjVoAoGCCqGSM49\n' +
      'AwEHoUQDQgAEurOxfSxmqIRYzJVagdZfMMSjRNNhB8i3mXyIMq704m2m52FdfKZ2\n' +
      'pQhByd5eyj3lgZ7m7jbchtdgyOF8Io/1ng==\n' +
      '-----END EC PRIVATE KEY-----\n';
    crypto.createSign('RSA-SHA256')._handle.sign(crypto._toBuf(private_key), 'ucs2');

I'm not sure if this was intentional for sign.sign() (to be always called with
null encoding), but I suspect not.

+@mlfbrown for joint work.