Enabling Additional Configuration Options in the Eucalyptus Elastic Load Balancer to Help with Troubleshooting

Follow

Eucalyptus Versions:  3.3.0 and Greater

Additional Configuration Options in the Eucalyptus Elastic Load Balancer to Help with Troubleshooting

This article will demonstrate how to enable additional configuration options on deployed Eucalyptus Load Balancer in order to help cloud administrators do the following:

  • Triage connection issues between the Eucalyptus Load Balancer and instances that are registered with the Eucalyptus Load Balancer
  • Enable additional logging output for the load-balancer-servo application to help with troubleshooting

This article can be used with the following knowledge base articles to help provide additional information to Support and/or in the Eucalyptus Jira Ticket System to aid in resolving issues with the Eucalyptus Elastic Load Balancer:

Prerequisites 

In order to enable these configuration options, the cloud administrator needs to have set the cloud property loadbalancing.loadbalancer_vm_keyname with a keypair created by a user under the 'eucalyptus' account.  For additional information, please refer to the section entitled 'Elastic Load Balancing' in the Eucalyptus Troubleshooting Guide

Note: In order to access any Eucalyptus Load Balancer, the cloud administrator must authorize port 22 (SSH) using euca-authorize with the security group associated with the load balancer.

Enable 'Debug' Logging - load-balancer-servo

The load-balancer-servo application is responsible for configuring HAProxy on a given Eucalyptus Load Balancer instance.  By default, it is set to use 'info' for logging to the /var/log/load-balancer-servo/servo.log file.  To change the option to 'debug', the cloud administrator needs to edit the /etc/init.d/load-balancer-servo script.  To test this out, SSH into a Eucalyptus Load Balancer instance:

[root@odc-f-13 ~]# euca-describe-instances
RESERVATION r-7204F61A 944786667073 euca-internal-408396244283-TestLoadBalancer
INSTANCE i-E4A82CCE emi-854029FD euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com euca-172-18-250-33.bigboi.internal running euca-elb 0 m1.medium 2014-07-09T15:14:21.908Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.233 172.18.250.33 instance-store hvm 0b329b96-c1b5-4e18-a211-7e63ff595c3c_ViciousLiesAndDangerousR_1 sg-012E35CD arn:aws:iam::944786667073:instance-profile/internal/loadbalancer/loadbalancer-vm-408396244283-TestLoadBalancer
TAG instance i-E4A82CCE Name loadbalancer-resources
TAG instance i-E4A82CCE aws:autoscaling:groupName asg-euca-internal-elb-408396244283-TestLoadBalancer
TAG instance i-E4A82CCE euca:node 10.105.10.7

[root@odc-f-13 ~]# ssh -i euca-elb.priv root@euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com

Change the LOGLEVEL value to 'debug':

[root@euca-172-18-250-33 ~]# grep LOGLEVEL= /etc/init.d/load-balancer-servo
LOGLEVEL=info
[root@euca-172-18-250-33 ~]# sed -i 's/LOGLEVEL=info/LOGLEVEL=debug/g' /etc/init.d/load-balancer-servo
[root@euca-172-18-250-33 ~]# grep LOGLEVEL= /etc/init.d/load-balancer-servo
LOGLEVEL=debug

Once that has been updated, reboot the Eucalyptus Load Balancer instance and observe the log level change in the /var/log/load-balancer-servo/servo.log file:

[root@euca-172-18-250-33 ~]# reboot

....

(After waiting a couple minutes, SSH back into the Eucalyptus Load Balancer instance, and review the load balancer log file)

[root@euca-172-18-250-33 ~]# tail -f /var/log/load-balancer-servo/servo.log
query string: AWSAccessKeyId=AKIF4ZEQUMDMTECIABY0&Action=PutServoStates&InstanceId=i-E4A82CCE&MetricData.member.1.MetricName=Latency&MetricData.member.1.Unit=Milliseconds&MetricData.member.1.Value=0&MetricData.member.2.MetricName=RequestCount&MetricData.member

.......

Method: GET
Path: /services/LoadBalancing/
Data:
Headers: {}
Host: loadbalancing.acme.eucalyptus-systems.com
Port: 8773
Params: {'InstanceId': 'i-E4A82CCE', 'Action': 'DescribeLoadBalancersByServo', 'Version': '2012-06-01'}
establishing HTTP connection: kwargs={'port': 8773, 'timeout': 70}
Token: ZXVjYQABAMbIDl3jpqwxkR4M4i6BWJX05c34GsnFj1eY52kCkTUSUdazX7Kx892mFkiqsnfGk0Kwjx+OZOuz8ndPDoRCpUs4F/Fbj9GWH6FN/ybjPxzS2zMSgoYMSVAl8eqc417nddO3VKmdm3K3CN5CtTQtJKNSQZ8wufU9ZHF7pZFNtczZXV3egPFhGrOSIHIswB2bXZabZc39ndP498A2KUaRNQb6iBU/99Q1cGVIOwr36++W1X83zp3c
using _calc_signature_2
query string: AWSAccessKeyId=AKIF4ZEQUMDMTECIABY0&Action=DescribeLoadBalancersByServo&InstanceId=i-E4A82CCE&SecurityToken=ZXVjYQABAMbIDl3jpqwxkR4M4i6BWJX05c34GsnFj1eY52kCkTUSUdazX7Kx892mFkiqsnfGk0Kwjx%2BOZOuz8ndPDoRCpUs4F%2FFbj9GWH6FN%2FybjPxzS2zMSgoYMSVAl8eqc417nddO3VKmdm3K3CN5CtTQtJKNSQZ8wufU9ZHF7pZFNtczZXV3egPFhGrOSIHIswB2bXZabZc39ndP498A2KUaRNQb6iBU%2F99Q1cGVIOwr36%2B%2BW1X83zp3c&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2014-07-10T18%3A47%3A13Z&Version=2012-06-01
string_to_sign: GET
loadbalancing.acme.eucalyptus-systems.com

Setting the 'debug' option helps provide more information regarding the load balancer's communication with Eucalyptus CloudWatch, Eucalyptus Security Token Service (STS), and backend instances that have been registered with the load balancer.

Enable 'stats socket' - HAProxy Configuration

The Eucalyptus Load Balancer uses HAProxy to help implement the Eucalyptus Elastic Load Balancing service.  HAProxy has a cool option to gather information about the application through a unix socket. The socket supports two modes:

  • interactive
  • non-interactive

The default is non-interactive.  By using the socat tool, the cloud administrator can send the socket commands that will interact with the HAProxy application.  

To get started, SSH into the Eucalyptus Load Balancer as performed previously.  After SSHing into the instance, install the socat rpm package:

[root@euca-172-18-250-33 ~]# yum install http://apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS/socat-1.7.2.4-1.el6.rf.x86_64.rpm

Once the software is installed, edit the /etc/load-balancer-servo/haproxy_template.conf file by adding the following entries under the 'global' section:

stats socket /var/run/haproxy.sock mode 600 level admin
stats timeout 2m

The 'global' section should look similar to the following after the edit:

global
 maxconn 100000
 ulimit-n 655360
 pidfile /var/run/haproxy.pid
 stats socket /var/run/haproxy.sock mode 600 level admin
 stats timeout 2m

 #drop privileges after port binding
 user servo
 group servo

After making the edit, reboot the instance as mentioned above, then SSH back into the instance.  The cloud administrator can then use either non-interactive or interactive operations with the /var/run/haproxy.sock socket.

Non-Interactive

To use the non-interactive method, use the 'echo' command to pass the desired operation.  For example, to show the information and statistics of the load balancer, do the following:

[root@euca-172-18-250-33 ~]# echo "show info;show stat" | socat unix-connect:/var/run/haproxy.sock stdio
Name: HAProxy
Version: 1.5-dev21-6b07bf7
Release_date: +2013/12/17
Nbproc: 1
Process_num: 1
Pid: 969
Uptime: 0d 5h17m02s
Uptime_sec: 19022
Memmax_MB: 0
Ulimit-n: 655360
Maxsock: 200032
Maxconn: 100000
Hard_maxconn: 100000
Maxpipes: 0
CurrConns: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 6
Run_queue: 1
Idle_pct: 100
node: euca-172-18-250-33.bigboi.inter
description:

# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,
HAProxy-Statistics,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,
HAProxy-Statistics,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,19022,0,,1,2,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,
http-80,FRONTEND,,,0,0,2000,0,0,0,0,0,0,,,,,OPEN,,,,,,,,,1,3,0,,,,0,0,0,0,,,,0,0,0,0,0,0,,0,0,0,,,0,0,0,0,
backend-http-80,BACKEND,0,0,0,0,200,0,0,0,0,0,,0,0,0,0,UP,0,0,0,,0,19022,0,,1,4,0,,0,,1,0,,0,,,,0,0,0,0,0,0,,,,,0,0,0,0,0,0,

Interactive

To interact directly with the socket, just use the socat command with the readline option.  Here is an example:

[root@euca-172-18-250-33 ~]# socat unix-connect:/var/run/haproxy.sock readline

prompt

> show info
Name: HAProxy
Version: 1.5-dev21-6b07bf7
Release_date: +2013/12/17
Nbproc: 1
Process_num: 1
Pid: 969
Uptime: 0d 5h19m54s
Uptime_sec: 19194
Memmax_MB: 0
Ulimit-n: 655360
Maxsock: 200032
Maxconn: 100000
Hard_maxconn: 100000
Maxpipes: 0
CurrConns: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
Tasks: 6
Run_queue: 1
Idle_pct: 100
node: euca-172-18-250-33.bigboi.inter
description:

> quit

If the load balancer is running on a Eucalyptus 4.0 cloud, and sticky sessions are involved, the 'show sess' operation will display all the known sessions.  To see the last known errors reported by the frontend and backend, use the 'show errors' operation.

Conclusion

Enabling debug logging for the load-balancer-servo application and enabling the unix socket connection to the HAProxy application can help provide more detailed information when triaging the Eucalyptus Load Balancer.  Not only it help when there is a bug with the load balancer, but it also helps the cloud administrator help the cloud user understand if they have misconfigured something with the backend application(s) running on instances that are registered with the load balancer.

Have more questions? Submit a request

Comments

Powered by Zendesk