Protobuf and lib conflicts: how to use gRPC with HBase

I’m a huge fan a gRPC. Really. I’ve talk about it some months (years now…) ago, and for now, it met all my needs: high-performance, light, well-structured, simple… and an active community behind it. Even Netflix, one of major pro-REST approach advocate in the open source community, began the switch to gRPC the last year and place Ribbon, their huge client side IPC library, in maintenance mode. And the ecosystem still grow: Nginx recently annonced a native support for gRPC traffic as well as the support for HTTP/2 server push in their 1.3.10 release.

NOTE

gRPC servers and clients use a binary encoding on the wire. So they are basically impossible to interact with using regular curl (and of course older versions of curl that do not support HTTP/2 are non-starters). grpcurl is a tool accepting messages using JSON encoding, which is much more friendly for both humans and scripts. Have a look here: https://github.com/fullstorydev/grpcurl

In one of my recent project, I have to use both the HBase client and a gRPC client (like one of the Google Apis) in a Java application. Everything was fine until I saw some errors due to version conflict on protobuf. Indeed, gRPC uses protobuf-java 3.5.1 via the dependency io.grpc:grpc-protobuf 1.13.1 and Hbase uses protobuf-java 2.5.0 via the the dependency org.apache.hbase:hbase-protocol. Let’s just use a recent version of protobuf (3.5.1) and see what happen, maybe? Nice try, but no…

Why? We always talk about backwards-compatibility in protobuf, so where is the problem? With protobuf, you can add new fields to your message formats without breaking backwards-compatibility: this is one of the pillar of protobuf: you can extend your protocol without having to worry about breaking existing code, your old binaries simply ignore the new field when parsing.

In the version 3 release, Google introduces a new language version – [https://developers.google.com/protocol-buffers/docs/overview#introducing-proto3](Protocol Buffers language version 3) (aka proto3).
Despite the fact that the two language version APIs are not completely compatible (they can possibly work indeed), the Java binaries for the two versions can not co-exist together. I would have preferred that they used a different package name (for example com.google.protobuf.v3) than the previous versions, allowing it to be used at the same time as an earlier version, like Apache Commons team did when released the Commons Lang 3.0, but unfortunately it’s not the case.

Well… There are not proper solution for this and if we want to fix it, there is one possible workaround using uber-jar and relocation. The idea is the following: we isolate the gRPC client and/or server generated code in one Maven module; inside this module, we configure Maven to generate the stubs and the messages classes, compile them, package them in jar and finally glue all the protobuf classes fetched from the gRPC dependencies in an other package name inside a new jar.

All this stuff can be handle with the the Apache Maven Shade Plugin. This plugin provides the capability to package the artifact in an uber-jar, including its dependencies and to shade – i.e. rename – the packages of some of the dependencies. Moreover, the Shade Plugin can relocate the classes which get included in the shaded artifact in order to create a private copy of their bytecode.

Take this project for our example:

├── my-awesome-multimode-project-with-grpc
│   ├── grpc-stuff
│   │   ├── src
│   │   │   └── main
│   │   │       └── proto
│   │   │           └── helloworld.proto
│   │   └── pom.xml
│   └── my-app
│       ├── src
│       │   └── main
│       │       └── java...
│       └── pom.xml
└── pom.xml

The grpc-stuff Maven module contains a pom.xml and a proto file definition using two plugin:
– the proto compiler plugin
– The Maven Shade Plugin move classes from the package com.google.protobuf and its subpackages into the package fr.layer4.myproject.protobuf by moving the corresponding JAR file entries and rewritting the affected bytecode.

Here is an extract of the pom.xml:

<project>
  ...
  <build>
    <extensions>
      <extension>
         <groupId>kr.motd.maven</groupId>
         <artifactId>os-maven-plugin</artifactId>
         <version>1.6.0</version>
      </extension>
   </extensions>
   <plugins>
      <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-compiler-plugin</artifactId>
      </plugin>
      <plugin>
         <groupId>org.xolstice.maven.plugins</groupId>
         <artifactId>protobuf-maven-plugin</artifactId>
         <configuration>
            <protocArtifact>com.google.protobuf:protoc:3.5.0:exe:${os.detected.classifier}</protocArtifact>
            <pluginId>grpc-java</pluginId>
            <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.13.1:exe:${os.detected.classifier}</pluginArtifact>
            <protoSourceRoot>src/main/proto</protoSourceRoot>
            <includes>
               <include>helloworld-v1.0.0.proto</include>
               <include>helloworld-v2.0.0.proto</include>
            </includes>
         </configuration>
         <executions>
            <execution>
               <goals>
                  <goal>compile</goal>
                  <goal>compile-custom</goal>
               </goals>
            </execution>
         </executions>
      </plugin>
      <plugin>
         <groupId>org.codehaus.mojo</groupId>
         <artifactId>build-helper-maven-plugin</artifactId>
         <executions>
            <execution>
               <id>add-classes</id>
               <phase>generate-sources</phase>
               <goals>
                  <goal>add-source</goal>
               </goals>
               <configuration>
                  <sources>
                     <source>${project.build.directory}/generated-sources</source>
                  </sources>
               </configuration>
            </execution>
         </executions>
      </plugin>
      <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-shade-plugin</artifactId>
         <executions>
            <execution>
               <phase>package</phase>
               <goals>
                  <goal>shade</goal>
               </goals>
            </execution>
         </executions>
         <configuration>
            <filters>
               <filter>
                  <artifact>*:*</artifact>
                  <excludes>
                     <exclude>META-INF/*.SF</exclude>
                     <exclude>META-INF/*.DSA</exclude>
                     <exclude>META-INF/*.RSA</exclude>
                  </excludes>
               </filter>
            </filters>
            <shadedClassifierName>proto3</shadedClassifierName>
            <shadedArtifactAttached>true</shadedArtifactAttached>
            <artifactSet>
               <includes>
                  <include>com.google.protobuf:*</include>
                  <include>io.grpc:grpc-protobuf</include>
                  <include>io.grpc:grpc-protobuf-lite</include>
               </includes>
            </artifactSet>
            <relocations>
               <relocation>
                  <pattern>com.google.protobuf</pattern>
                  <shadedPattern>my.package.protobuf</shadedPattern>
               </relocation>
               <relocation>
                  <pattern>io.grpc.protobuf</pattern>
                  <shadedPattern>my.package.grpc</shadedPattern>
               </relocation>
            </relocations>
         </configuration>
      </plugin>
    </plugins>
  </build>
  ...
</project>

By default, this plugin will replace the project’s main artifact with the shaded artifact. If you have multiple clients module using gRPC which are not all the time used with other dependencies based on protobuf (like HBase), you can keep both the original and the shaded artifact in the repository, and configure the plugin to attach the shaded artifact as a secondary artifact:

<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>proto3</shadedClassifierName>

Happy coding!

Sources:
– http://www.grpc.io/blog/principles
– https://developers.google.com/protocol-buffers/docs/overview
– https://maven.apache.org/plugins/maven-shade-plugin/index.html

Credits:
“File:Wireshark – TCP.png” by JackPotte is licensed under CC BY-SA 3.0 / Resized

Related Posts

Leave a comment

About privacy:

This site uses Akismet to reduce spam. Learn how your comment data is processed.