A Simple OpenPGP API

In this post I want to share how easy it is to use OpenPGP using the Stateless OpenPGP Protocol (SOP).

I talked about the SOP specification and its purpose and benefits already in past blog posts. This time I want to give some in-depth examples of how the API can be used in your application.

There are SOP API implementations available in different languages like Java and Rust. They have in common, that they are based around the Stateless OpenPGP Command Line Specification, so they are very similar in form and function.

For Java-based systems, the SOP API was defined in the sop-java library. This module merely contains interface definitions. It is up to the user to choose a library that provides an implementation for those interfaces. Currently the only known implementation is pgpainless-sop based on PGPainless.

The single entry point to the SOP API is the SOP interface (obviously). It provides methods for OpenPGP actions. All we need to get started is an instantiation of this interface:

// This is an ideal candidate for a dependency injection framework!
SOP sop = new SOPImpl(); // provided by pgpainless-sop

Let’s start by generating a secret key for the user Alice:

byte[] key = sop.generateKey()
        .userId("Alice <alice@example.org>")
        .generate()
        .getBytes();

The resulting byte array now contains our OpenPGP secret key. Next, lets extract the public key certificate, so that we can share it with out contacts.

// public key
byte[] cert = sop.extractCert()
        .key(key) // secret key
        .getBytes();

There we go! Both byte arrays contain the key material in ASCII armored form (which we could disable by calling .noArmor()), so we can simply share the certificate with our contacts.

Let’s actually create an encrypted, signed message. We obviously need our secret key from above, as well as the certificate of our contact Bob.

// get bobs certificate
byte[] bobsCert = ...

byte[] message = "Hello, World!\n".getBytes(StandardCharsets.UTF_8);

byte[] encryptedAndSigned = sop.encrypt()
        .signWith(key) // sign with our key
        .withCert(cert) // encrypt for us, so that we too can decrypt
        .withCert(bobsCert) // encrypt for Bob
        .plaintext(message)
        .getBytes();

Again, by default this message is ASCII armored, so we can simply share it as a String with Bob.

We can decrypt and verify Bobs reply like this:

// Bobs answer
byte[] bobsEncryptedSignedReply = ...

ByteArrayAndResult<DecryptionResult> decrypted = sop.decrypt()
        .verifyWithCert(bobsCert) // verify bobs signature
        .withKey(key) // decrypt with our key
        .ciphertext(bobsEncryptedSignedReply)
        .toByteArrayAndResult();

// Bobs plaintext reply
byte[] message = decrypted.getBytes();
// List of signature verifications
List<Verification> verifications = decrypted.getResult().getVerifications();

Easy! Signing messages and verifying signed-only messages basically works the same, so I’ll omit examples for it in this post.

As you can see, performing basic OpenPGP operations using the Stateless OpenPGP Protocol is as simple as it gets. And the best part is that the API is defined as an interface, so swapping the backend can simply be done by replacing the SOP object with an implementation from another library. All API usages stay the same.

I want to use this opportunity to encourage YOU the reader: If there is no SOP API available for your language of choice, consider creating one! Take a look at the specification and an API definition like sop-java or the sop Rust crate to get an idea of how to design the API. If you keep your SOP API independent from any backend library it will be easy to swap backends out for another library later in the process.

Let’s make OpenPGP great again! Happy Hacking!

Progress on PGPainless Development

Image of capsule tower in Japan

Not much time has passed since I last wrote about my progress on the PGPainless library. However, I feel like its time for an update.

Since the big 0.2.0 release, 4 further releases, 0.2.1 through 0.2.4 have been published. Taken together, the changes are quite substantial, so let me summarize.

Image of capsule tower in Japan
Photo by Raphael Koh on Unsplash

Modular SOP Implementation

The (in my opinion) most exciting change is that there now is an experimental module of java interfaces that model the Stateless OpenPGP Protocol (SOP). This module named sop-java is completely independent from PGPainless and has no external dependencies whatsoever. Its basically a port of Sequoia-PGP’s sop crate (which in term is based around the Stateless OpenPGP Command Line Interface specification) to Java.

Applications that want to execute basic OpenPGP operations can depend on this interface and decide on the concrete implementation later without locking themselves in with one fixed implementation. Remember:

The Database Is a Detail. […] The Web Is a Detail. […] Frameworks Are Details.

Uncle Bob – Clean Architecture

The module sop-java-picocli contains a CLI frontend for sop-java. It uses the picocli library to closely model the Stateless OpenPGP Command Line Interface (SOP-CLI) specification (version 1 for now).

The exciting part is that this module too is independent from PGPainless, but it can be used by any library that implements sop-java.

Next up, the contents of pgpainless-sop drastically changed. While up until recently it contained a fully fledged SOP-CLI application which used pgpainless-core directly, it now no longer contains command line application code, but instead an implementation of sop-java using pgpainless-core. Therefore pgpainless-sop can be used as a drop-in for sop-java, making it the first java-based SOP implementation (to my knowledge).

Lastly, pgpainless-cli brings sop-java-picocli and pgpainless-sop together. The code does little more than to plug pgpainless-sop as SOP backend into the command line application, resulting in a fully functional OpenPGP command line application (basically what pgpainless-sop was up until release 0.2.3, just better :P).

$ ./pgpainless-cli help
Usage: pgpainless-cli [COMMAND]
Commands:
  help          Displays help information about the specified command
  armor         Add ASCII Armor to standard input
  dearmor       Remove ASCII Armor from standard input
  decrypt       Decrypt a message from standard input
  encrypt       Encrypt a message from standard input
  extract-cert  Extract a public key certificate from a secret key from
                  standard input
  generate-key  Generate a secret key
  sign          Create a detached signature on the data from standard input
  verify        Verify a detached signature over the data from standard input
  version       Display version information about the tool

The exciting part about this modular design is that if YOU are working on an OpenPGP library for Java, you don’t need to re-implement a CLI frontend on your own. Instead, you can implement the sop-java interface and benefit from the CLI provided by sop-java-picocli for free.

If you are a library consumer, depending on sop-java instead of pgpainless-core would allow you to swap out PGPainless for another library, should any emerge in the future. It also means that porting your application to other platforms and languages might become easier, thanks to the more or less fixed API provided by the SOP protocol.

Further Changes

There are some more exciting changes worth mentioning.

The whole PGPainless suite can now be built reproducibly!

$ gradle --quiet clean build &> /dev/null && md5sum {.,pgpainless-core,pgpainless-sop,pgpainless-cli,sop-java,sop-java-picocli}/build/libs/*.jar

e7e9f45eb9d74540092920528bb0abf0  ./build/libs/PGPainless-0.2.4.jar
8ab68285202c8a303692c7332d15c2b2  pgpainless-core/build/libs/pgpainless-core-0.2.4.jar
a9c1d7b4a47d5ec66fc65131c14f4848  pgpainless-sop/build/libs/pgpainless-sop-0.2.4.jar
08cfb620a69015190e45d66548b8ea0f  pgpainless-cli/build/libs/pgpainless-cli-0.2.4.jar
e309d5a8d3a9439c6fae1c56150d9d07  sop-java/build/libs/sop-java-0.2.4.jar
9901849535f57f04b615afb06216ae5c  sop-java-picocli/build/libs/sop-java-picocli-0.2.4.jar

It actually was not hard at all to achieve reproducibility. The command line application has a version command, which extracted the current application version by accessing a version.properties file which would be written during the Gradle build.

Unfortunately, Java’s implementation of the Properties class includes a timestamp when writing out the object into a PrintStream. Therefore the result was not reproducible. The fix was to write the file manually, without using a Properties object.

Furthermore, the APIs for decryption/verification were further simplified, following the example of the encryption API. Instead of chained builder subclasses, there now is a single builder class which is used to receive decryption keys and public key certificates etc.

If you need more details about what changed in PGPainless, there now is a changelog file.

PGPainless 0.2 Released!

I’m very proud and excited to announce the release of PGPainless version 0.2! Since the last stable release of my OpenPGP library for Java and Android 9 months ago, a lot has changed and improved! Most importantly development on PGPainless is being financially sponsored by FlowCrypt, so I was able to focus a lot more energy into working on the library. I’m very grateful for this opportunity ๐Ÿ™‚

A red letter, sealed with a wax seal
Photo by Natasya Chen on Unsplash

PGPainless is using Bouncycastle, but aims to save developers from the pain of writing lots of boilerplate code, while at the same time using the BC API properly. The new release is available on maven central, the source code can be found on Github and Codeberg.

PGPainless is now depending on Bouncycastle 1.68 and the minimum Android API level has been raised to 10 (Android 2.3.3). Let me bring you up to speed about some of its features and the recent changes!

Inspect Keys!

Back in the last stable release, PGPainless could already be used to generate keys. It offered some shortcut methods to quickly generate archetypal keys, such as simple RSA keys, or key rings based on elliptic curves. In version 0.2, support for additional key algorithms was added, such as EdDSA or XDH.

The new release introduces PGPainless.inspectKeyRing(keyRing) which allows you to quickly access information about a key, such as its user-ids, which subkeys are encryption capable and which can be used to sign data, their expiration dates, algorithms etc.

Furthermore this feature can be used to evaluate a key at a certain point in time. That way you can quickly check, which key flags or algorithm preferences applied to the key 3 weeks ago, when that key was used to create that signature you care about. Or you can check, which user-ids your key had 5 years ago.

Edit Keys!

Do you already have a key, but want to extend its expiration date? Do you have a new Email address and need to add it as a user-id to your key? PGPainless.modifyKeyRing(keyRing) allows basic modification of a key. You can add additional user-ids, adopt subkeys into your key, or expire/revoke existing subkeys.

secretKeys = PGPainless.modifyKeyRing(secretKeys)
                       .setExpirationDate(expirationDate, keyRingProtector)
                       .addSubKey(subkey, subkeyProtector, keyRingProtector)
                       .addUserId(UserId.onlyEmail("alice@pgpainless.org"), keyRingProtector)
                       .deleteUserId("alice@pgpainful.org", keyRingProtector)
                       .revokeSubkey(subKeyId, keyRingProtector)
                       .revokeUserId("alice@pgpaintrain.org", keyRingProtector)
                       .changePassphraseFromOldPassphrase(oldPass)
                           .withSecureDefaultSettings().toNewPassphrase(newPass)
                       .done();

Encrypt and Sign Effortlessly!

PGPainless 0.2 comes with an improved, simplified encryption/signing API. While the old API was already quite intuitive, I was focusing too much on the code being “autocomplete-friendly”. My vision was that the user could encrypt a message without ever having to read a bit of documentation, simply by typing PGPainless and then following the autocomplete suggestions of the IDE. While the result was successful in that, the code was not very friendly to bind to real-world applications, as there was not one builder class, but several (one for each “step”). As a result, if a user wanted to configure the encryption dynamically, they would have to keep track of different builder objects and cope with casting madness.

// Old API
EncryptionStream encryptionStream = PGPainless.createEncryptor()
        .onOutputStream(targetOuputStream)
        .toRecipient(aliceKey)
        .and()
        .toRecipient(bobsKey)
        .and()
        .toPassphrase(Passphrase.fromPassword("password123"))
        .usingAlgorithms(SymmetricKeyAlgorithm.AES_192, HashAlgorithm.SHA256, CompressionAlgorithm.UNCOMPRESSED)
        .signWith(secretKeyDecryptor, aliceSecKey)
        .asciiArmor();

Streams.pipeAll(plaintextInputStream, encryptionStream);
encryptionStream.close();

The new API is still intuitive, but at the same time it is flexible enough to be modified with future features. Furthermore, since the builder has been divided it is now easier to integrate PGPainless dynamically.

// New shiny 0.2 API
EncryptionStream encryptionStream = PGPainless.encryptAndOrSign()
        .onOutputStream(outputStream)
        .withOptions(
                ProducerOptions.signAndEncrypt(
                        new EncryptionOptions()
                                .addRecipient(aliceKey)
                                .addRecipient(bobsKey)
                                // optionally encrypt to a passphrase
                                .addPassphrase(Passphrase.fromPassword("password123"))
                                // optionally override symmetric encryption algorithm
                                .overrideEncryptionAlgorithm(SymmetricKeyAlgorithm.AES_192),
                        new SigningOptions()
                                // Sign in-line (using one-pass-signature packet)
                                .addInlineSignature(secretKeyDecryptor, aliceSecKey, signatureType)
                                // Sign using a detached signature
                                .addDetachedSignature(secretKeyDecryptor, aliceSecKey, signatureType)
                                // optionally override hash algorithm
                                .overrideHashAlgorithm(HashAlgorithm.SHA256)
                ).setAsciiArmor(true) // Ascii armor
        );

Streams.pipeAll(plaintextInputStream, encryptionStream);
encryptionStream.close();

Verify Signatures Properly!

The biggest improvement to PGPainless 0.2 is improved, proper signature verification. Prior to this release, PGPainless was doing what probably every other Bouncycastle-based OpenPGP library was doing:

PGPSignature signature = [...];
// Initialize the signature with the public signing key
signature.init(pgpContentVerifierBuilderProvider, signingKey);

// Update the signature with the signed data
int read;
while ((read = signedDataInputStream.read()) != -1) {
    signature.update((byte) read);
}

// Verify signature correctness
boolean signatureIsValid = signature.verify();

The point is that the code above only verifies signature correctness (that the signing key really made the signature and that the signed data is intact). It does however not check if the signature is valid.

Signature validation goes far beyond plain signature correctness and entails a whole suite of checks that need to be performed. Is the signing key expired? Was it revoked? If it is a subkey, is it bound to its primary key correctly? Has the primary key expired or revoked? Does the signature contain unknown critical subpackets? Is it using acceptable algorithms? Does the signing key carry the SIGN_DATA flag? You can read more about why signature verification is hard in my previous blog post.

After implementing all those checks in PGPainless, the library now scores second place on Sequoia’s OpenPGP Interoperability Test Suite!

Lastly, support for verification of cleartext-signed messages such as emails was added.

New SOP module!

Also included in the new release is a shiny new module: pgpainless-sop

This module is an implementation of the Stateless OpenPGP Command Line Interface specification. It basically allows you to use PGPainless as a command line application to generate keys, encrypt/decrypt, sign and verify messages etc.

$ # Generate secret key
$ java -jar pgpainless-sop-0.2.0.jar generate-key "Alice <alice@pgpainless.org>" > alice.sec
$ # Extract public key
$ java -jar pgpainless-sop-0.2.0.jar extract-cert < alice.sec > alice.pub
$ # Sign some data
$ java -jar pgpainless-sop-0.2.0.jar sign --armor alice.sec < message.txt > message.txt.asc
$ # Verify signature
$ java -jar pgpainless-sop-0.2.0.jar verify message.txt.asc alice.pub < message.txt 
$ # Encrypt some data
$ java -jar pgpainless-sop-0.2.0.jar encrypt --sign-with alice.sec alice.pub < message.txt > message.txt.asc
$ # Decrypt ciphertext
$ java -jar pgpainless-sop-0.2.0.jar decrypt --verify-with alice.pub --verify-out=verif.txt alice.sec < message.txt.asc > message.txt

The primary reason for creating this module though was that it enables PGPainless to be plugged into the interoperability test suite mentioned above. This test suite uncovered a ton of bugs and shortcomings and helped me massively to understand and interpret the OpenPGP specification. I can only urge other developers who work on OpenPGP libraries to implement the SOP specification!

Upstreamed changes

Even if you are not yet convinced to switch to PGPainless and want to keep using vanilla Bouncycastle, you might still benefit from some bugfixes that were upstreamed to Bouncycastle.

Every now and then for example, BC would fail to do some internal conversions of elliptic curve encryption keys. The source of this issue was that BC was converting keys from BigIntegers to byte arrays, which could be of invalid length when the encoding was having leading zeros, thus omitting one byte. Fixing this was easy, but finding the bug was taking quite some time.

Another bug caused decryption of messages which were encrypted for more than one key/passphrase to fail, when BC tried to decrypt a Symmetrically Encrypted Session Key Packet with the wrong key/passphrase first. The cause of this issue was that BC was not properly rewinding the decryption stream after reading a checksum, thus corrupting decryption for subsequent attempts with the correct passphrase or key. The fix was to mark and rewind the stream properly before the next decryption attempt.

Lastly some methods in BC have been modernized by adding generics to Iterators. Chances are if you are using BC 1.68, you might recognize some changes once you bump the dependency to BC 1.69 (once it is released of course).

Thank you!

I would like to thank anyone who contributed to the new release in any way or form for their support. Special thanks goes to my sponsor FlowCrypt for giving me the opportunity to spend so much time on the library! Furthermore I’d like to thank the all the amazing folks over at Sequoia-PGP for their efforts of improving the OpenPGP ecosystem and patiently helping me understand the (at times at bit muddy) OpenPGP specification.

PGPainless 0.1.0 released

Image of a lock

After two years and a dozen alpha versions I am very glad to announce the first stable release of PGPainless! The release is available on maven central.

PGPainless aims to make using OpenPGP with Bouncycastle fun again by abstracting away most of the complexity and overhead that normally comes with it. At the same time PGPainless remains configurable by making heavy use of the builder pattern for almost everything.

Lets take a look at how to create a fresh OpenPGP key:

        PGPKeyRing keyRing = PGPainless.generateKeyRing()
                .simpleEcKeyRing("alice@wonderland.lit", "password123");

That is all it takes to generate an OpenPGP keypair that uses ECDH+ECDSA keys for encryption and signatures! You can of course also configure a more complex key pair with different algorithms and attributes:

        PGPainless.generateKeyRing()
                .withSubKey(KeySpec.getBuilder(RSA_ENCRYPT.withLength(RsaLength._4096))
                        .withKeyFlags(KeyFlag.ENCRYPT_COMMS, KeyFlag.ENCRYPT_STORAGE)
                        .withDefaultAlgorithms())
                .withSubKey(KeySpec.getBuilder(ECDH.fromCurve(EllipticCurve._P256))
                        .withKeyFlags(KeyFlag.ENCRYPT_COMMS, KeyFlag.ENCRYPT_STORAGE)
                        .withDefaultAlgorithms())
                .withSubKey(KeySpec.getBuilder(RSA_SIGN.withLength(RsaLength._4096))
                        .withKeyFlags(KeyFlag.SIGN_DATA)
                        .withDefaultAlgorithms())
                .withMasterKey(KeySpec.getBuilder(RSA_SIGN.withLength(RsaLength._8192))
                        .withKeyFlags(KeyFlag.CERTIFY_OTHER)
                        .withDetailedConfiguration()
                        .withPreferredSymmetricAlgorithms(SymmetricKeyAlgorithm.AES_256)
                        .withPreferredHashAlgorithms(HashAlgorithm.SHA512)
                        .withPreferredCompressionAlgorithms(CompressionAlgorithm.BZIP2)
                        .withFeature(Feature.MODIFICATION_DETECTION)
                        .done())
                .withPrimaryUserId("alice@wonderland.lit")
                .withPassphrase(new Passphrase("password123".toCharArray()))
                .build();

The API is designed in a way so that the user can very hardly make mistakes. Inputs are typed, so that as an example the user cannot input a wrong key length for an RSA key. The “shortcut” methods (eg. withDefaultAlgorithms()) uses sane, secure defaults.

Now that we have a key, lets encrypt some data!

        byte[] secretMessage = message.getBytes(UTF8);
        ByteArrayOutputStream envelope = new ByteArrayOutputStream();

        EncryptionStream encryptor = PGPainless.createEncryptor()
                .onOutputStream(envelope)
                .toRecipients(recipientPublicKey)
                .usingSecureAlgorithms()
                .signWith(keyDecryptor, senderSecretKey)
                .asciiArmor();

        Streams.pipeAll(new ByteArrayInputStream(secretMessage), encryptor);
        encryptor.close();
        byte[] encryptedSecretMessage = envelope.toByteArray();

As you can see there is almost no boilerplate code! At the same time, above code will create a stream that will encrypt and sign all the data that is passed through. In the end the envelope stream will contain an ASCII armored message that can only be decrypted by the intended recipients and that is signed using the senders secret key.

Decrypting data and/or verifying signatures works very similar:

        ByteArrayInputStream envelopeIn = new ByteArrayInputStream(encryptedSecretMessage);
        DecryptionStream decryptor = PGPainless.createDecryptor()
                .onInputStream(envelopeIn)
                .decryptWith(keyDecryptor, recipientSecretKey)
                .verifyWith(senderPublicKey)
                .ignoreMissingPublicKeys()
                .build();

        ByteArrayOutputStream decryptedSecretMessage = new ByteArrayOutputStream();

        Streams.pipeAll(decryptor, decryptedSecretMessage);
        decryptor.close();
        OpenPgpMetadata metadata = decryptor.getResult();

The decryptedSecretMessage stream now contains the decrypted message. The metadata object can be used to get information about the message, eg. which keys and algorithms were used to encrypt/sign the data and if those signatures were valid.

In summary, PGPainless is now able to create different types of keys, read encrypted and unencrypted keys, encrypt and/or sign data streams as well as decrypt and/or verify signatures. The latest additions to the API contain support for creating and verifying detached signatures.

PGPainless is already in use in Smacks OpenPGP module which implements XEP-0373: OpenPGP for XMPP and it has been designed primarily with the instant messaging use case in mind. So if you want to add OpenPGP support to your application, feel free to give PGPainless a try!

How to Implement a XEP for Smack.

Photo of the Earth taken from Space

Smack is a FLOSS XMPP client library for Java and Android app development. It takes away much of the burden a developer of a chat application would normally have to carry, so the developer can spend more time working on nice stuff like features instead of having to deal with the protocol stack.

Many (80+ and counting) XMPP Extension Protocols (XEPs) are already implemented in Smack. Today I want to bring you along with me and add support for one more.

What Smack does very well is to follow the Open-Closed-Principle of software architecture. That means while Smacks classes are closed for modification by the developer, it is pretty easy to extend Smack to add support for custom features. If Smack doesn’t fit your needs, don’t change it, extend it!

The most important class in Smack is probably the XMPPConnection, as this is where messages coming from and going to. However, even more important for the developer is what is being sent.

XMPP’s strength comes from the fact that arbitrary XML elements can be exchanged by clients and servers. Heck, the server doesn’t even have to understand what two clients are sending each other. That means that if you need to send some form of data from one device to another, you can simply use XMPP as the transport protocol, serialize your data as XML elements with a namespace that you control and send if off! It doesn’t matter, which XMPP server software you choose, as the server more or less just forwards the data from the sender to the receiver. Awesome!

So lets see how we can extend Smack to add support for a new feature without changing (and therefore potentially breaking) any existing code!

For this article, I chose XEP-0428: Fallback Indication as an example protocol extension. The goal of Fallback Indication is to explicitly mark <body/> elements in messages as fallback. For example some end-to-end encryption mechanisms might still add a body with an explanation that the message is encrypted, so that older clients that cannot decrypt the message due to lack of support still display the explanation text instead. This enables the user to switch to a better client ๐Ÿ˜› Another example would be an emoji in the body as fallback for a reaction.

XEP-0428 does this by adding a fallback element to the message:

<message from="alice@example.org" to="bob@example.net" type="chat">
  <fallback xmlns="urn:xmpp:fallback:0"/>  <-- THIS HERE
  <encrypted xmlns="urn:example:crypto">Rgreavgl vf abg n irel ybat
gvzr nccneragyl.</encrypted>
  <body>This message is encrypted.</body>
</message>

If a client or server encounter such an element, they can be certain that the body of the message is intended to be a fallback for legacy clients and act accordingly. So how to get this feature into Smack?

After the XMPPConnection, the most important types of classes in Smack are the ExtensionElement interface and the ExtensionElementProvider class. The later defines a class responsible for deserializing or parsing incoming XML into the an object of the former class.

The ExtensionElement is itself an empty interface in that it does not provide anything new, but it is composed from a hierarchy of other interfaces from which it inherits some methods. One notable super class is NamedElement, more on that in just a second. If we start our XEP-0428 implementation by creating a class that implements ExtensionElement, our IDE would create this class body for us:

package tk.jabberhead.blog.wow.nice;

import org.jivesoftware.smack.packet.ExtensionElement;
import org.jivesoftware.smack.packet.XmlEnvironment;

public class FallbackIndicationElement implements ExtensionElement {
    
    @Override
    public String getNamespace() {
        return null;
    }

    @Override
    public String getElementName() {
        return null;
    }

    @Override
    public CharSequence toXML(XmlEnvironment xmlEnvironment) {
        return null;
    }
}

The first thing we should do is to change the return type of the toXML() method to XmlStringBuilder, as that is more performant and gains us a nice API to work with. We could also leave it as is, but it is generally recommended to return an XmlStringBuilder instead of a boring old CharSequence.

Secondly we should take a look at the XEP to identify what to return in getNamespace() and getElementName().

<fallback xmlns="urn:xmpp:fallback:0"/>
[   ^    ]      [        ^          ]
element name          namespace

In XML, the part right after the opening bracket is the element name. The namespace follows as the value of the xmlns attribute. An element that has both an element name and a namespace is called fully qualified. That’s why ExtensionElement is inheriting from FullyQualifiedElement. In contrast, a NamedElement does only have an element name, but no explicit namespace. In good object oriented manner, Smacks ExtensionElement inherits from FullyQualifiedElement which in term is inheriting from NamedElement but also introduces the getNamespace() method.

So lets turn our new knowledge into code!

package tk.jabberhead.blog.wow.nice;

import org.jivesoftware.smack.packet.ExtensionElement;
import org.jivesoftware.smack.packet.XmlEnvironment;

public class FallbackIndicationElement implements ExtensionElement {
    
    @Override
    public String getNamespace() {
        return "urn:xmpp:fallback:0";
    }

    @Override
    public String getElementName() {
        return "fallback";
    }

    @Override
    public XmlStringBuilder toXML(XmlEnvironment xmlEnvironment) {
        return null;
    }
}

Hm, now what about this toXML() method? At this point it makes sense to follow good old test driven development practices and create a JUnit test case that verifies the correct serialization of our element.

package tk.jabberhead.blog.wow.nice;

import static org.jivesoftware.smack.test.util.XmlUnitUtils.assertXmlSimilar;
import org.jivesoftware.smackx.pubsub.FallbackIndicationElement;
import org.junit.jupiter.api.Test;

public class FallbackIndicationElementTest {

    @Test
    public void serializationTest() {
        FallbackIndicationElement element = new FallbackIndicationElement();

        assertXmlSimilar("<fallback xmlns=\"urn:xmpp:fallback:0\"/>",
element.toXML());
    }
}

Now we can tweak our code until the output of toXml() is just right and we can be sure that if at some point someone starts messing with the code the test will inform us of any breakage. So what now?

Well, we said it is better to use XmlStringBuilder instead of CharSequence, so lets create an instance. Oh! XmlStringBuilder can take an ExtensionElement as constructor argument! Lets do it! What happens if we return new XmlStringBuilder(this); and run the test case?

<fallback xmlns="urn:xmpp:fallback:0"

Almost! The test fails, but the builder already constructed most of the element for us. It prints an opening bracket, followed by the element name and adds an xmlns attribute with our namespace as value. This is typically the “head” of any XML element. What it forgot is to close the element. Lets see… Oh, there’s a closeElement() method that again takes our element as its argument. Lets try it out!

<fallback xmlns="urn:xmpp:fallback:0"</fallback>

Hm, this doesn’t look right either. Its not even valid XML! (ใƒŽเฒ ็›Šเฒ )ใƒŽๅฝกโ”ปโ”โ”ป Normally you’d use such a sequence to close an element which contained some child elements, but this one is an empty element. Oh, there it is! closeEmptyElement(). Perfect!

<fallback xmlns="urn:xmpp:fallback:0"/>
package tk.jabberhead.blog.wow.nice;

import org.jivesoftware.smack.packet.ExtensionElement;
import org.jivesoftware.smack.packet.XmlEnvironment;

public class FallbackIndicationElement implements ExtensionElement {
    
    @Override
    public String getNamespace() {
        return "urn:xmpp:fallback:0";
    }

    @Override
    public String getElementName() {
        return "fallback";
    }

    @Override
    public XmlStringBuilder toXML(XmlEnvironment xmlEnvironment) {
        return new XmlStringBuilder(this).closeEmptyElement();
    }
}

We can now serialize our ExtensionElement into valid XML! At this point we could start sending around FallbackIndications to all our friends and family by adding it to a message object and sending that off using the XMPPConnection. But what is sending without receiving? For this we need to create an implementation of the ExtensionElementProvider custom to our FallbackIndicationElement. So lets start.

package tk.jabberhead.blog.wow.nice;

import org.jivesoftware.smack.packet.XmlEnvironment;
import org.jivesoftware.smack.provider.ExtensionElementProvider;
import org.jivesoftware.smack.xml.XmlPullParser;

public class FallbackIndicationElementProvider
extends ExtensionElementProvider<FallbackIndicationElement> {
    
    @Override
    public FallbackIndicationElement parse(XmlPullParser parser,
int initialDepth, XmlEnvironment xmlEnvironment) {
        return null;
    }
}

Normally implementing the deserialization part in form of a ExtensionElementProvider is tiring enough for me to always do that last, but luckily this is not the case with Fallback Indications. Every FallbackIndicationElement always looks the same. There are no special attributes or – shudder – nested named child elements that need special treating.

Our implementation of the FallbackIndicationElementProvider looks simply like this:

package tk.jabberhead.blog.wow.nice;

import org.jivesoftware.smack.packet.XmlEnvironment;
import org.jivesoftware.smack.provider.ExtensionElementProvider;
import org.jivesoftware.smack.xml.XmlPullParser;

public class FallbackIndicationElementProvider
extends ExtensionElementProvider<FallbackIndicationElement> {
    
    @Override
    public FallbackIndicationElement parse(XmlPullParser parser,
int initialDepth, XmlEnvironment xmlEnvironment) {
        return new FallbackIndicationElement();
    }
}

Very nice! Lets finish the element part by creating a test that makes sure that our provider does as it should by creating another JUnit test. Obviously we have done that before writing any code, right? We can simply put this test method into the same test class as the serialization test.

    @Test
    public void deserializationTest()
throws XmlPullParserException, IOException, SmackParsingException {
        String xml = "<fallback xmlns=\"urn:xmpp:fallback:0\"/>";
        FallbackIndicationElementProvider provider =
new FallbackIndicationElementProvider();
        XmlPullParser parser = TestUtils.getParser(xml);

        FallbackIndicationElement element = provider.parse(parser);

        assertEquals(new FallbackIndicationElement(), element);
    }

Boom! Working, tested code!

But how does Smack learn about our shiny new FallbackIndicationElementProvider? Internally Smack uses a Manager class to keep track of registered ExtensionElementProviders to choose from when processing incoming XML. Spoiler alert: Smack uses Manager classes for everything!

If we have no way of modifying Smacks code base, we have to manually register our provider by calling

ProviderManager.addExtensionProvider("fallback", "urn:xmpp:fallback:0",
new FallbackIndicationElementProvider());

Element providers that are part of Smacks codebase however are registered using an providers.xml file instead, but the concept stays the same.

Now when receiving a stanza containing a fallback indication, Smack will parse said element into an object that we can acquire from the message object by calling

FallbackIndicationElement element = message.getExtension("fallback",
"urn:xmpp:fallback:0");

You should have noticed by now, that the element name and namespace are used and referred to in a number some places, so it makes sense to replace all the occurrences with references to a constant. We will put these into the FallbackIndicationElement where it is easy to find. Additionally we should provide a handy method to extract fallback indication elements from messages.

...

public class FallbackIndicationElement implements ExtensionElement {
    
    public static final String NAMESPACE = "urn:xmpp:fallback:0";
    public static final String ELEMENT = "fallback";

    @Override
    public String getNamespace() {
        return NAMESPACE;
    }

    @Override
    public String getElementName() {
        return ELEMENT;
    }

    ...

    public static FallbackIndicationElement fromMessage(Message message) {
        return message.getExtension(ELEMENT, NAMESPACE);
    }
}

Did I say Smack uses Managers for everything? Where is the FallbackIndicationManager then? Well, lets create it!

package tk.jabberhead.blog.wow.nice;

import java.util.Map;
import java.util.WeakHashMap;

import org.jivesoftware.smack.Manager;
import org.jivesoftware.smack.XMPPConnection;

public class FallbackIndicationManager extends Manager {

    private static final Map<XMPPConnection, FallbackIndicationManager>
INSTANCES = new WeakHashMap<>();

    public static synchronized FallbackIndicationManager
getInstanceFor(XMPPConnection connection) {
        FallbackIndicationManager manager = INSTANCES.get(connection);
        if (manager == null) {
            manager = new FallbackIndicationManager(connection);
            INSTANCES.put(connection, manager);
        }
        return manager;
    }

    private FallbackIndicationManager(XMPPConnection connection) {
        super(connection);
    }
}

Woah, what happened here? Let me explain.

Smack uses Managers to provide the user (the developer of an application) with an easy access to functionality that the user expects. In order to use some feature, the first thing the user does it to acquire an instance of the respective Manager class for their XMPPConnection. The returned instance is unique for the provided connection, meaning a different connection would get a different instance of the manager class, but the same connection will get the same instance anytime getInstanceFor(connection) is called.

Now what does the user expect from the API we are designing? Probably being able to send fallback indications and being notified whenever we receive one. Lets do sending first!

    ...

    private FallbackIndicationManager(XMPPConnection connection) {
        super(connection);
    }

    public MessageBuilder addFallbackIndicationToMessage(
MessageBuilder message, String fallbackBody) {
        return message.setBody(fallbackBody)
                .addExtension(new FallbackIndicationElement());
}

Easy!

Now, in order to listen for incoming fallback indications, we have to somehow tell Smack to notify us whenever a FallbackIndicationElement comes in. Luckily there is a rather nice way of doing this.

    ...

    private FallbackIndicationManager(XMPPConnection connection) {
        super(connection);
        registerStanzaListener();
    }

    private void registerStanzaListener() {
        StanzaFilter filter = new AndFilter(StanzaTypeFilter.MESSAGE, 
                new StanzaExtensionFilter(FallbackIndicationElement.ELEMENT, 
                        FallbackIndicationElement.NAMESPACE));
        connection().addAsyncStanzaListener(stanzaListener, filter);
    }

    private final StanzaListener stanzaListener = new StanzaListener() {
        @Override
        public void processStanza(Stanza packet) 
throws SmackException.NotConnectedException, InterruptedException,
SmackException.NotLoggedInException {
            Message message = (Message) packet;
            FallbackIndicationElement fallbackIndicator =
FallbackIndicationElement.fromMessage(message);
            String fallbackBody = message.getBody();
            onFallbackIndicationReceived(message, fallbackIndicator,
fallbackBody);
        }
    };

    private void onFallbackIndicationReceived(Message message,
FallbackIndicationElement fallbackIndicator, String fallbackBody) {
        // do something, eg. notify registered listeners etc.
    }

Now that’s nearly it. One last, very important thing is left to do. XMPP is known for its extensibility (for the better or the worst). If your client supports some feature, it is a good idea to announce this somehow, so that the other end knows about it. That way features can be negotiated so that the sender doesn’t try to use some feature that the other client doesn’t support.

Features are announced by using XEP-0115: Entity Capabilities, which is based on XEP-0030: Service Discovery. Smack supports this using the ServiceDiscoveryManager. We can announce support for Fallback Indications by letting our manager call

ServiceDiscoveryManager.getInstanceFor(connection)
        .addFeature(FallbackIndicationElement.NAMESPACE);

somewhere, for example in its constructor. Now the world knows that we know what Fallback Indications are. We should however also provide our users with the possibility to check if their contacts support that feature as well! So lets add a method for that to our manager!

    public boolean userSupportsFallbackIndications(EntityBareJid jid) 
            throws XMPPException.XMPPErrorException,
SmackException.NotConnectedException, InterruptedException, 
            SmackException.NoResponseException {
        return ServiceDiscoveryManager.getInstanceFor(connection())
                .supportsFeature(jid, FallbackIndicationElement.NAMESPACE);
    }

Done!

I hope this little article brought you some insights into the XMPP protocol and especially into the development process of protocol libraries such as Smack, even though the demonstrated feature was not very spectacular.

Quick reminder that the next Google Summer of Code is coming soon and the XMPP Standards Foundation got accepted ๐Ÿ˜‰
Check out the project ideas page!

Happy Hacking!

Dealing with Colors in lower Android versions

I’m currently working on a project which requires me to dynamically set the color of certain UI elements to random RGB values.

Unfortunately and surprisingly, handy methods for dealing with colors in Android are only available since API level 26 (Android O). Luckily though, the developer reference specifies, how colors are encoded internally in the Android operating system, so I was able to create a class with the most important color-related (for my use-case) methods which is able to function in lower Android versions as well.

Link to the gist

I hope I can save some of you from headaches by sharing the code. Feel free to reuse as you please ๐Ÿ™‚

Happy Hacking!

PS: Is there a way to create Github-like gists with Gitea?

QR-Code Generator for OMEMO

OMEMO is, like any other encryption protocol based on trust. The user has to make sure, that the keys they are trusting belong to the right users. Otherwise a so called Man-in-the-Middle attack is possible. An attacker might pretend to be your contact and secretly steal all the messages you thought were encrypted. They are, just to the wrong recipient.

To counteract such attacks, OMEMO encourages the user to verify their contacts fingerprints. A fingerprint can be considered the name of a contacts key. The user has to make sure, that the key A he is presented with really belongs to contact C by checking, if the fingerprints match. As those fingerprints are usually long, random series of characters, this is a tedious task. Luckily there are techniques like QR codes, which make our lifes easier. Instead of comparing two long strings character by character, you just scan a code and are done.

The QR-Code contains the Jabber-ID of the owner, as well as all of their fingerprints. So by scanning your code, a friend might automatically add you to their contact list and mark your devices as trusted. You can also put the QR-Code on your personal website, so people who want to reach out to you can easily establish a secure connection to you.

I spent the last few days looking into JavaFX (I have no real UI designing experience in Java, apart from Android), designing a small tool that can generate OMEMO fingerprint QR-Codes. This is what I came up with:

QR-Code generator with selectable fingerprints

The tool logs into your XMPP account and fetches all your published keys. Then it presents you with a list in which you can select, which fingerprints you want to include in the QR-Code.

There are still a lot of features missing and I consider the tool in no means as completed. My plans are to add the possibility to export QR-Codes to file, as well as to copy the text content of the code to clipboard. You see, there is a lot of work left to do, however I wanted to share my thoughts with you, so maybe client developers can adopt my idea.