Java

#

23 Feb: Testing BigData projects

Writing tests that use a traditional database is hard. But writing tests in a project using Hadoop is really harder. Hadoop stacks are complex pieces of software and if you want to test your Hadoop projects, it may be a real nightmare: – many components are involved, you are not just using HBase, but HBase, Zookeeper and a DFS. – a lot of configuration is needed – cleaning the data of the previous tests relies on many…

#

15 Feb: Fitting Java and Python with JPY

There are many libraries in Java (more than 176,649 unique artifacts indexed just on Maven Central), but sometimes you can not find what you are looking for, except for a Python equivalent. In a previous project, I had to deal with custom MaxMind databases. Maxmind provides a Java library with a database reader, but does not provides a database writer. After some researches, I found one official lib in Perl, and an other (unofficial) in Python. Since,…

#

06 Dec: Knox in production: avoid pitfalls and common mistakes

I’ve already post articles about Knox some weeks ago about two subjects: how to use the HBase REST API througth Knox and how to submit Spark job via the Knox API. In my current mission, many projects are now using Knox as main gateway for many services like HBase and HDFS, but also for Oozie, Yarn… After some weeks of development and deployment in production, I’ve decided to write a post about some troubles that you may…

#

25 Oct: Efficient logging with Spring Boot, Logback and Logstash

Logging is an important part of any entreprise application and Logback makes an excellent choice: it’s simple, fast, light and very powerful. Spring Boot has a great support for Logback and provides lot of features to configure it. In this article I will present you an integration for an entreprise logging stack using Logback, Spring Boot and Logstash. WARNING The Spring Boot recommands to use the -spring variants for your logging configuration (for example logback-spring.xml rather than…

#

17 Oct: Advanced tools: playing with Java Native Access

This post results from a recent deep diving in the source code of Elasticsearch, which uses JNA mainly for memory management when configuring the mlockall. I will present you how to use JNA in a very simple example: how to check the user who has launched the JVM. Note Remember that when you are thinking about solution using OS native calls, you must deal and depend with platform librairies. So use it carefully and only for specific…

12 May: JSON pretty print with… Spring Boot

I wrote a post a few months ago about a pretty-print hack for JAX-RS (https://layer4.fr/2015/02/json-pretty-print-with-jax-rsjackson/). Here is a version using Spring Boot. All the stuff is located in a @Configuration class extending WebMvcConfigurerAdapter: import java.io.IOException; import java.util.List; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Configuration; import org.springframework.http.converter.HttpMessageConverter; import org.springframework.http.converter.json.MappingJackson2HttpMessageConverter; import org.springframework.web.context.request.RequestAttributes; import org.springframework.web.context.request.RequestContextHolder; import org.springframework.web.context.request.ServletRequestAttributes; import org.springframework.web.servlet.config.annotation.EnableWebMvc; import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter; import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jackson.core.util.DefaultPrettyPrinter; import com.fasterxml.jackson.databind.ObjectMapper; @Configuration @EnableWebMvc public class WebConfiguration extends WebMvcConfigurerAdapter { @Autowired private ObjectMapper mapper; @Override public void extendMessageConverters(List<HttpMessageConverter<?>>…

23 Feb: JSON pretty print with JAX-RS/Jackson

If you have already worked with ElasticSearch, you may have used a very useful companion when dealing with your curl: the “?pretty” parameter which allows you to see the JSON response well-formated. If not, have a look here, you’ll thank me later: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/common-options.html It can be usefull too to have this kind of feature in your JAX-RS project. And this is “pretty” simple (…). All you have to do is create a class extending JacksonJsonProvider: import java.io.IOException;…

07 Feb: High performance RSS/Atom parsing

Parsing RSS feeds is very easy in Java. Several libs exist to get the job done: feed4j, rssowl, Apache Abdera and many others. But the most commly used is ROME. ROME is a set of RSS and Atom Utilities for Java. It makes it easy to work in Java with most syndication formats: RSS 0.9x, 1.0, 2.0 and Atom 0.3, 1.0. Reading RSS from a source is dead-simple, you need these dependencies: <!– Rome Atom+RSS –> <dependency>…

04 Nov: Pooling a Thrift client

Thrift is an interface definition language that is used to define and create services for numerous languages. The Thrift stack relies on protocols (TBinaryProtocol, TCompactProtocol…) and transports (TSocket, TFileTransport…). But, since the transport layer is essentially a wrapper on a socket or a file, Thrift is NOT thread-safe. Like other resources not thread-safe, you have the choice: work with a costly locking algo, create each time a new connection to the resource or think about pool your…

21 Oct: A best Spring AsyncRestTemplate!

I wrote an article on the famous Spring’s RestTemplate some months ago (https://layer4.fr/2013/02/24/a-best-spring-resttemplate/). Since the last version of Spring 4, currently 4.0.0.M3, or the BUILD-SNAPSHOT for the most courageous, you can use a custom RestTemplate in an async way: the AsyncRestTemplate API. Why should I use that? The purpose of this API is to allow Java applications to easily execute HTTP requests and asynchronously process the HTTP responses. In non async mode, the code will block until…