Sun Java memory fun with logstash

I am working on a distributed logstash deployment in AWS.
I'm using the Elasticcache-Redis provided by AWS as the store between the syslog receiver and the elasticcache writer (aka the worker).
I keep getting OOM errors on the worker like this
{:timestamp=>"2013-10-22T08:18:58.592000+0000", :message=>"Failed to flush outgoing items", :outgoing_count=>34, :exception=>java.lang.OutOfMemoryError: Java heap space, :backtrace=>[], :level=>:warn}

I was aware of the ability to change the heapsize allocated to the jvm, but wasn't sure how to find it.  Luckily google helped.  I took the command line I was using to run elasticsearch and compared how the 3 settings affected the defaults

java -Xms512m -Xmx512m -Xss256k -XX:+PrintFlagsFinal -version > /tmp/1
java -XX:+PrintFlagsFinal -version > /tmp/2
diff /tmp/1

That helped me figure out which default settings were important.

Now, I just have to figure out how to give logstash enough RAM to not die.

Comments

  1. aShok.......October 23, 2013 at 1:10:00 AM CDT

    Hi,

    I got such results:

    # diff /tmp/1 /tmp/2
    253c253
    < uintx InitialHeapSize := 536870912 {product}
    ---
    > uintx InitialHeapSize := 527180032 {product}
    296c296
    < uintx MaxHeapSize := 536870912 {product}
    ---
    > uintx MaxHeapSize := 8436842496 {product}
    523c523
    < intx ThreadStackSize := 256 {pd product}
    ---
    > intx ThreadStackSize = 1024 {pd product}

    So based on the above diff - how much memory should I allocate?

    Very useful post!

    Thanks
    Ashok

    ReplyDelete

Post a Comment

Popular posts from this blog

Xubuntu Home Server on Dell XPS 13 9370

[ General warning, this isn't a newbie or step-by-step instruction on how to do this.  If you follow this blindly or don't know what a command does you might wipe something value on your system] Introduction I picked up this box through the Dell Outlet for working remotely, but not at home.  It functions okay for that task but I'm in need of a new home server to replace my old Alienware 14 R2 ancient box. Introduction Before Xubuntu 20.04 Installation BIOS Changes Make Bitwise Factory Backup Fresh Xubuntu Installation Zero out the harddrive Install Xubuntu Install latest updates System Configuration LUKS + PAM home directory Boot Xubuntu Live Image & Prepare new partition Create new user with admin privileges Configure PAM mount for primary user Install Software APT Repository Non APT Golang Google Chrome Configuration BIND PIN Entry Nginx + GIT docker XPS 13 9370 Clickpad Buttons Postfix via Gmail ZRam RamDisk Rsyslog Before Xubuntu 20.04 Installation BIOS Changes Acce
Read more

Cygwin + syslog-ng

Until I can get a real workstation at work and get off this winblows, Cygwin is my friend.  I am working on some OO programming with PERL and really need to be able to log syslog messages locally. After installing syslog-ng, I couldn't figure out how to run it, till I stumbled on a page that mentioned the syslogd-config command. Initially I had some problems because the /etc/passwd and /etc/group files were not world readable (don't know if that is a side effect of the Legato restore of my laptop or just the default behavior) however a quick " chmod a+r /etc/passwd /etc/group " seemed to help. running syslogd-config yielded this output: > syslogd-ng-config Creating default /etc/syslog-ng.conf file Warning: The syslogd service is already installed.  You can not run both, syslogd and syslog-ng in parallel. Do you want to deinstall the syslogd service in favor of syslog-ng? (yes/no) yes Warning: The following function requires administrator privileg
Read more

Fedora - VNC through systemd/xinetd

Configuring Fedora 17 for Xvnc through systemd/xinetd I found a number of links around but nothing exactly covered my setup.  I wanted to be able to run an X11 session from home as securely and efficiently as possible.  After configuring my home server like this: sudo yum install xinetd tigervnc-server-minimal sudoedit /etc/gdm/custom.conf [security] RelaxPermissions=0 [xdmcp] Enable=1 HonorIndirect=1 [greeter] Quiver=1 sudo useradd -d /var/empty/vncuser -m -s /bin/false vncuser sudoedit /etc/services vnc-1280 5910/tcp # VNC Listener sudoedit /etc/xinetd.d/vnc # BEGIN service vnc-1280 { socket_type = stream protocol = tcp wait = no user = vncuser server = /usr/bin/Xvnc server_args = -inetd -query localhost -desktop 1024x768x24 -once -geometry 1024x768 -NeverShared securitytypes=none } # END sudo systemctl start xinetd.service And you will need to restart GDM.  I couldn't get the systemctl restart prefdm stuff quite right, so I rebooted.  (Yeah,
Read more