Skip to main content

Centralize logs with logstash

Now a days logging is the essential part of the any application. Logging useful piece of information can easily help to find errors, fixing bug and much more. Modern application now scaling up in hundreds servers in cloud. Managing and monitoring logs of heterogeneous system is very challenging for any system administrator even more challenging for developers to fixing bugs. Last Friday evening we stared our testing with 3rd party products and stuck with a few bugs. As usual we first looked for the log and tried to find some hints to reproduce the bug. Here we got serious problems, our application scaled on few application servers such as Oracle GlassFish, Apache Tomcat. It was not a pleasant moments to search all over the server to find a few pieces of information. Here we have understand that we have to use any tools to manage, monitor and search logs. With my experience we have a few options:
1) Flume, Hadoop hdfs and ElasticSearch.
2) Kafka, Storm and Slor.
3) Logstash and Graylog2.
First option with Hadoop we have implemented in few cases but for the current project it seems very big gun. Second option also need some configuration and coding experience to build up the log managements tools from the scratch. My aim was to use something new and elegant which we can configured with less effort and easy to use. A few times i heard about logstash and decided to make a try. In the rest of the post i will describe how to install and configure logstash for centralizing log, i.e. collecting, aggregating and searching log. Most of the features of logstash is as follows:
1) Collecting log through agents
2) Aggregating logs
3) Shipping the logs in ElasticSearch
4) Web interface for searching logs
5) Open source
6) Everything in an one jar, nothing more.
7) Very well documented with examples

Take a look at the high level architecture of logstash:
For centralizing you have to need the followings components:
1) ElasticSearch
2) Redis
I have go through the getting started page and everything runs fines as a charm. Only one error i have got when tried to install Redis.
$ make
clang: error: no such file or directory: '../deps/hiredis/libhiredis.a'
clang: error: no such file or directory: '../deps/lua/src/liblua.a'
make[1]: *** [redis-server] Error 1
make: *** [all] Error 2
By googling in internet i have found the solution very easily as follows:
$ make
cd deps
make lua hiredis linenoise
and finalized the installation 
$ make
cd $REDIS_CODE/src
make
In my cases i wanted to collect log from the Glassfish server.log and use the basic configuration for agent
input {
    file {
    type => "server"

    # Wildcards work, here :)
    path => [ "$DOMAIN_HOME/logs/*.log" ]
  }
}

output {
  #stdout { codec => rubydebug }
  redis { host => "127.93.1.11" data_type => "list" key => "crm" }
}
That's all. Happy coding and bloging.
Labels:

Comments

Post a Comment

Popular posts from this blog

Send e-mail with attachment through OSB

Oracle Service Bus (OSB) contains a good collection of adapter to integrate with any legacy application, including ftp, email, MQ, tuxedo. However e-mail still recognize as a stable protocol to integrate with any application asynchronously. Send e-mail with attachment is a common task of any business process. Inbound e-mail adapter which, integrated with OSB support attachment but outbound adapter doesn't. This post is all about sending attachment though JavaCallout action. There are two ways to handle attachment in OSB: 1) Use JavaCallout action to pass the binary data for further manipulation. It means write down a small java library which will get the attachment and send the e-mail. 2) Use integrated outbound e-mail adapter to send attachment, here you have to add a custom variable named attachment and assign the binary data to the body of the attachment variable. First option is very common and easy to implement through javax.mail api, however a much more developer manage t
Post a Comment
Read more

Tip: SQL client for Apache Ignite cache

A new SQL client configuration described in  The Apache Ignite book . If it got you interested, check out the rest of the book for more helpful information. Apache Ignite provides SQL queries execution on the caches, SQL syntax is an ANSI-99 compliant. Therefore, you can execute SQL queries against any caches from any SQL client which supports JDBC thin client. This section is for those, who feels comfortable with SQL rather than execute a bunch of code to retrieve data from the cache. Apache Ignite out of the box shipped with JDBC driver that allows you to connect to Ignite caches and retrieve distributed data from the cache using standard SQL queries. Rest of the section of this chapter will describe how to connect SQL IDE (Integrated Development Environment) to Ignite cache and executes some SQL queries to play with the data. SQL IDE or SQL editor can simplify the development process and allow you to get productive much quicker. Most database vendors have their own front-en
Post a Comment
Read more

Load balancing and fail over with scheduler

Every programmer at least develop one Scheduler or Job in their life time of programming. Nowadays writing or developing scheduler to get you job done is very simple, but when you are thinking about high availability or load balancing your scheduler or job it getting some tricky. Even more when you have a few instance of your scheduler but only one can be run at a time also need some tricks to done. A long time ago i used some data base table lock to achieved such a functionality as leader election. Around 2010 when Zookeeper comes into play, i always preferred to use Zookeeper to bring high availability and scalability. For using Zookeeper you have to need Zookeeper cluster with minimum 3 nodes and maintain the cluster. Our new customer denied to use such a open source product in their environment and i was definitely need to find something alternative. Definitely Quartz was the next choose. Quartz makes developing scheduler easy and simple. Quartz clustering feature brings the HA and
Post a Comment
Read more