Graphite FreeBSD Quick

From Devpit
Jump to: navigation, search

Installation

Graphite UI:

  • pkg install py27-graphite-web
  • Edit: /usr/local/etc/carbon/carbon.conf
  • Edit: /usr/local/etc/carbon/storage-schemas.conf
  • Edit: /usr/local/etc/graphite/local_settings.py
  • cp /usr/local/etc/graphite/graphite.wsgi.example /usr/local/etc/graphite/graphite.wsgi

Apache to serve it:

  • pkg install apache24 ap24-mod_wsgi3
  • Create: /usr/local/etc/apache24/Includes/graphite.conf

Startup:

  • Add to /etc/rc.conf:
apache24_enable="YES"
carbon_enable="YES"
carbon_logdir="/graphite/carbon/log"
carbon_user="www"
  • vipw: change www to /var/empty
  • Run:
service carbon start
mkdir -p /graphite/carbon/log /graphite/carbon/rrd /graphite/carbon/whisper /graphite/log
chown -Rvv www:www /graphite
(cd /usr/local/lib/python2.7/site-packages/graphite && sudo -Eu www python2 manage.py syncdb)
service apache24 start

Now you should be able to load Graphite in a browser.


carbon/carbon.conf

Adjust as follows:

-#LOCAL_DATA_DIR = /opt/graphite/storage/whisper/
-LOCAL_DATA_DIR = /var/db/carbon/whisper/
+GRAPHITE_ROOT = /grapite
+GRAPHITE_CONF_DIR = /usr/local/etc/carbon
+GRAPHITE_STORAGE_DIR = /graphite/carbon/
+STORAGE_DIR = /graphite/carbon/
+LOCAL_DATA_DIR = /graphite/carbon/whisper/
+CONF_DIR = /usr/local/etc/carbon
+LOG_DIR = /graphite/carbon/log/
+PID_DIR = /var/run
-USER =
+USER = www


carbon/storage-schemas.conf

Adjust as follows:

-[default_1min_for_1day]
+[default]
-retentions = 60s:1d
+retentions = 60s:1827d


graphite/local_settings.py

Adjust as follows:

-#SECRET_KEY = 'UNSAFE_DEFAULT'
+SECRET_KEY = 'type any string'
-#GRAPHITE_ROOT = '/opt/graphite'
+GRAPHITE_ROOT = '/graphite'
-STORAGE_DIR = '/var/db/carbon'
+STORAGE_DIR = '/graphite/carbon/'
-WHISPER_DIR = '/var/db/carbon/whisper'
+WHISPER_DIR = '/graphite/carbon/whisper/'
-RRD_DIR = '/var/db/carbon/rrd'
+RRD_DIR = '/graphite/carbon/rrd'
-LOG_DIR = '/var/log/graphite'
+LOG_DIR = '/graphite/log'
-INDEX_FILE = '/var/db/graphite/index'
+INDEX_FILE = '/graphite/carbon/index'
-'NAME': '/var/db/graphite/graphite.db',
+'NAME': '/graphite/graphite.db',


apache24/Includes/graphite.conf

Create as follows, refer to /usr/local/share/graphite-web/examples/example-graphite-vhost.conf:

<IfModule !wsgi_module.c>
    LoadModule wsgi_module modules/mod_wsgi.so
</IfModule>

WSGISocketPrefix /var/run/wsgi

<VirtualHost *:80>
        ServerName graphite
        DocumentRoot "/usr/local/share/graphite-web/content"

        # I've found that an equal number of processes & threads tends
        # to show the best performance for Graphite (ymmv).
        WSGIDaemonProcess graphite processes=5 threads=5 display-name='%{GROUP}' inactivity-timeout=120
        WSGIProcessGroup graphite
        WSGIApplicationGroup %{GLOBAL}
        WSGIImportScript /usr/local/etc/graphite/graphite.wsgi process-group=graphite application-group=%{GLOBAL}

        WSGIScriptAlias / /usr/local/etc/graphite/graphite.wsgi

        Alias /content/ /usr/local/share/graphite-web/content/
        <Location "/content/">
                SetHandler None
        </Location>
        <Directory "/usr/local/share/graphite-web/content">
		Require all granted
        </Directory>

        # XXX In order for the django admin site media to work you
        # must change @DJANGO_ROOT@ to be the path to your django
        # installation, which is probably something like:
        # /usr/lib/python2.6/site-packages/django
        Alias /media/ "@DJANGO_ROOT@/contrib/admin/media/"
        <Location "/media/">
                SetHandler None
        </Location>

        <Directory "/usr/local/etc/graphite">
		Require all granted
        </Directory>
</VirtualHost>


Clients

  • pkg install collectd5
  • Edit /usr/local/etc/collectd.conf: enable write_graphite and other plugins

Now you should see data start arriving in the UI.


collectd.conf

Overwrite with:

#Hostname "localhost"
FQDNLookup false
BaseDir "/var/db/collectd"
PIDFile "/var/run/collectd.pid"
PluginDir "/usr/local/lib/collectd"
TypesDB "/usr/local/share/collectd/types.db"
#AutoLoadPlugin false
Interval 10
Timeout 2
ReadThreads 2
WriteThreads 1
#WriteQueueLimitHigh 1000000
#WriteQueueLimitLow   800000

# Plugins which provide logging functions should be loaded first.
LoadPlugin syslog
<Plugin syslog>
	LogLevel info
</Plugin>

LoadPlugin write_graphite
<Plugin write_graphite>
	<Node "graphite">
		Host "graphite"
		Port "2003"
		Protocol "tcp"
		LogSendErrors true
		#Prefix "x."
		#Postfix ".y"
		EscapeCharacter "_"
		StoreRates false
		SeparateInstances true
		AlwaysAppendDS true
	</Node>
</Plugin>

LoadPlugin contextswitch
LoadPlugin cpu
LoadPlugin load
LoadPlugin memory
LoadPlugin swap
LoadPlugin zfs_arc
LoadPlugin ntpd

<Plugin ntpd>
	ReverseLookups false
</Plugin>


carbon-aggregator

Characters like parentheses will work for collecting data, but Graphite's UI will choke. (It's already been opened as a bug and claimed fixed, but not.) You'll probably want to just ax any unusual characters to avoid problems now or later. However to do this you need to pipe everything through carbon-aggregator.py; rewrite-rules.conf is not handled by carbon-cache.py. In /usr/local/etc/carbon/rewrite-rules.conf, add the following. Surprisingly, the last line will work even though it seems to conflict with the config file syntax. (As you probably know by now, Graphite is pretty great but also rather sloppy.) For fun I included a few other clean-up rules.

[pre]
\.snmp\. = .
\.ping\.ping$ = .ping.rtt
\.ping\.ping_droprate$ = .ping.loss
\.ping\.ping_stddev$ = .ping.rtt_stddev
- = _
[^A-Za-z0-9_\.\:] =

In /usr/local/etc/carbon/carbon.conf under [aggregator], add:

USER = www

In /usr/local/etc/carbon/carbon.conf under [cache], change:

-LINE_RECEIVER_INTERFACE = 0.0.0.0
+LINE_RECEIVER_INTERFACE = 127.0.0.1
-UDP_RECEIVER_INTERFACE = 0.0.0.0
+UDP_RECEIVER_INTERFACE = 127.0.0.1
-PICKLE_RECEIVER_INTERFACE = 0.0.0.0
+PICKLE_RECEIVER_INTERFACE = 127.0.0.1

Change all your collectd.conf and other feeds to send data to port 2023/2024 instead of 2003/2004.

Create the missing rc.d script and add carbon_aggregator_enable=YES to /etc/rc.conf. A decent rc.d/carbon_aggregator modeled after rc.d/carbon would look like:

#!/bin/sh

# PROVIDE: carbon_aggregator
# KEYWORD: shutdown

# 
# Add the following lines to /etc/rc.conf.local or /etc/rc.conf
# to enable this service:
#
# carbon_aggregator_enable (bool):	Set to NO by default.
# carbon_conf, carbon_user, carbon_group, carbon_debug, carbon_logdir: set from rc.d/carbon

. /etc/rc.subr

name=carbon_aggregator
rcvar=carbon_aggregator_enable

load_rc_config ${name}

: ${carbon_aggregator_enable:=NO}
: ${carbon_user:=root}
: ${carbon_group:=wheel}
: ${carbon_conf:=/usr/local/etc/carbon/carbon.conf}
: ${carbon_debug:=NO}
: ${carbon_logdir:=/var/db/carbon/log/}
: ${pidfile:=/var/run/carbon_aggregator.pid}

stop_cmd="${name}_stop"
required_files="${carbon_conf} /usr/local/etc/carbon/storage-schemas.conf"

command_interpreter="/usr/local/bin/python2.7"
command="/usr/local/bin/carbon-aggregator.py"
command_args="--config=${carbon_conf} --logdir ${carbon_logdir} --pidfile ${pidfile} start"

carbon_aggregator_stop()
{
    echo "Stopping $name"
    pids=`check_pidfile ${pidfile} ${command} ${command_interpreter}`

    /usr/local/bin/carbon-aggregator.py --config=${carbon_conf} --pidfile ${pidfile} stop
    wait_for_pids ${pids}
}

run_rc_command "$1"

Finally, restart carbon-cache with "service carbon restart" and start carbon-aggregator with "service carbon_aggregator start".

After you see things are working, you may want to rename any old .wsp files onto the new names to avoid starting new timelines. Merely renaming the files is all there is to it.

More notes

  • manage.py works with the file specified by graphite/local_settings.py in DATABASES/'default'/'NAME' (above suggestion is /graphite/graphite.db).
  • In the above configuration, all mutable data is in /graphite, so if you need to start over just blow that away and run the mkdir and chown and manage.py commands above.
  • collectd5 can query SNMP servers, but FreeBSD's default options for the package don't add the SNMP plugin. You may want to rebuild it for that, or you can use the exec plugin with snmpwalk/snmpget.
  • Graphana might be a nice alternative frontend.
  • In the whisper storage directory, you can delete unwanted metrics by merely removing the files. There doesn't seem to be any state stored elsewhere. If more data is received for a metric you removed, carbine-cache will quickly create a new one.
  • whisper-dump.py is a critical tool for debugging. You can see be sure metrics are arriving correctly and watch the settings in storage-aggregation.conf and storage-schemas.conf in action. If the UI isn't doing what you expect, this will help you reason about whether the problem is in the collection or the UI.