Most serious applications (and distributed microservices style architectures) will require to provide a log aggregation & analysis feature to its dev & operations teams. Reviewing log entires from 10s or 100s of server instances is not something to take lightly. Whether you choose to use a commercial product or an open source offering – that does not matter; just make sure you have one available.
Recently I have been deploying applications using AWS Beanstalk. You can definitely configure CloudWatch Logs to send log streams over to AWS ElasticSearch service. Log messages can be routed to a Lambda function which would break the log messages into individual attributes suitable for indexing. I wanted to try a slightly different route where I depend less on CloudWatch Logs and more on open source tools. Enter filebeat on Beanstalk.
Updated one of my previous Spring Boot sample service to run within a Docker container – https://github.com/thomasma/quote-service-docker. You can run it locally w/o Docker as a regular Spring Boot app and next run it inside a Docker container. Make sure that you have Docker setup correctly and tested prior to running this app.
To discuss Serverless Architecture we need to understand how we got here. From using physical machines we moved to virtual machines (somewhere in between a few brave folks also used linux/solaris containers). The current trend is container technologies such as Docker or CoreOS RKT which allow even more efficient use of resources. Regardless of which you use, we are often required to plan our application infrastructure needs upfront and permanently keep the “servers” running.
While SpringMVC makes it quite easy to create RESTful services, this starter project adds a few things more. It provides a consistent way to send error messages in json back to the caller and also integrates Spring Security into the mix.
Here is a slightly modified Architecture on my previous post on Getting Started With ElasticSearch. If you find yourself indexing content constantly (like 100s or even 1000s per minute) you might want to consider an asynchronous architecture towards indexing.
Integrating systems in a complex enterprise landscape can get tough. You have all kinds of interactions going from one system to the other. Many of them taking in and spitting out different data formats. Which means you have to not only worry about the routing between these integrations but also the transformations between them. Updated to use Camel 2.11., Spring 3.2.2 and ActiveMQ 5.7.0. Continue reading →
In Part-II of this series on Spring Batch, I went through an example of reading from a flat file and persisting into the database. In this article I will go through the reverse. Read 200,000 rows from the database and export it into a comma separated flat file. Continue reading →