%

Goaccess / OpenBSD

Article published the ; modified the
9 minutes to read

This article has 1900 words.
RAW source of the article:
Commit version: 36e3c52

Description

Goaccess is a FLoSS, known to be light, fast, in order to to analyze in real time or not the activity on a web server, either directly within a Unix terminal, or on the HTTPS protocol.

It’s able to produce statistics on HTML, JSON even CSV format.

⇒ Environnement:

  • OpenBSD: 6.9 → 7.1

Installation

Usual: # pkg_add goaccess

Configuration

The main file config is on: /etc/goaccess/goaccess.conf.

⇒ You can use goaccess without any modifications on its file config. This need to pass all options on the CLI, but few options are clearly commons and can be “frozen” on the file config.

⇒ Too, it’s also possible to create numerous files config, each specific to a web domain, to set particularly segun this domain. In this case, you need to specify it…

In this article, I use one configuration file, and put some options on the CLI.

Now, let’s see the main configuration options:

Time and Date formats

Formatting the time!

⇒ If you use httpd, you need to be careful to set correctly the style of the log, and into in the goaccess configuration file in the Time Format Options section.

⇒ For nginx, it will be enough to uncomment the following options:

  • time-format %H:%M:%S
  • date-format %d/%b/%Y - this being into Date Format Options section.

Log format

  • log-format: prefer the COMBINED format — for httpd, you can choose the common format, too.

If you use nginx, it is imperative to register all statuses, not to filter the 2xx and 3xx — if you had created the map directive you will have at least to comment it and restart the web service.

File Options

If you have only one web domaine on the server to analyze, it’s usefull to set the log-file option with the absolute directory of the access.log relient.

Parse Options

  • exclude-ip: use this option to exclude adresses IP or network segments, like you personal network at home

  • 444-as-404: the 444 error is nginx’s specific. If you want to analyze as an 404 error, change to true. It’s up to you!

  • ignore-crawlers: to ignore all crawlers robots; change to true

  • ignore-panel is one panel that can be disabled. On Europe, with the RGPD, it is better to disable, at least, the REMOTE_USER panel.

  • anonymize-ip : change to true - egual, because RGPD, disable!

Persistence Options

  • db-path: the absolute directory of the goaccess database ­— configure only one domain.
  • persist: to backup the analyzed datas.
  • restore: load the data to be visualized from the saved data.

⇒ Enable both last options to true, if you want to save the data into the goaccess database.

Of course, there are numerous options.

After, just use goaccess.

Now, let’s push the configuration a further:

User system

I prefer to create and use a dedicated user system. Among the advantages are the search in log messages and even cron.

# useradd -s /sbin/nologin -d /var/db/goaccess _goaccess

⇒ to search in log messages:

$ grep goaccess /var/log/messages or $ doas grep _goaccess /var/cron/log

doas

⇒ I preferred to add the permission to use the goaccess binary into /etc/doas.conf:

permit nopass _goaccess cmd /usr/local/bin/goaccess

(I’m not absolutly sure about the necessity).

database directory

⇒ Go to create the main directory for the goaccess database:

# mkdir -p /var/db/goaccess
# chown _goaccess:daemon /var/db/goaccess

(personally, I preferred another absolute folder).

Think to add at yours backups!

Then if like me, you have several web domains on your server, create many subdirectories with the name of the web domain, such as:

$ domain=
$ doas -u _goaccess mkdir "/var/db/goaccess/${domain}"

Thus, future statistics will be really dedicated to a domain.

crontab

⇒ You need to set cron rules:

$ doas -u _goaccess crontab -e

(Egual, it’s possible to do by: doas crontab -u _goaccess -e).

Add:

*/15 * * * * -ns goaccess -a --db-path "/var/db/goaccess/domain/" -f /var/www/logs/domain/access.log -o /var/www/goaccess/domain/stats.html

Littles explainations:

  • one crontab to execute every 15 minutes. Keep on mind, it’s an example.
  • replace the domain string by the domain name.

Authenticate

It’s up to you if you want web authentication before viewing; some people think you should, others don’t; personally I prefer it.

After using htpasswd, you need to configure on the server directive on the virtual host:

  • for http, use authenticate.
  • on nginx, both auth_basic and auth_basic_user_file directives.

httpd configuration

Basically:

server "domain.tld" {

(…)

	root "/htdocs/domain.tld/www"
	
	location "/stats" {
		authenticate with "/file_htpwd"
		directory auto index
	}

(…)

}

Do not forget: the path of the htpasswd file is relative to the web chroot!

nginx configuration

(…)

location /stats/ {
    auth_basic "Auth Area";
    auth_basic_user_file /file_htpwd;    
    autoindex on;
}

(…)

Voila for the “configuration” part!

Use

Basic use

Execute the binary, as:

$ goaccess -o /var/www/htdocs/domain.tld/stats/index.html

use _goaccess

⇒ Use goaccess with the _goaccess user:

For instance, the first time, to precise datation with month and year:

$ domain=
$ date="$(date +'%Y-%m')"
$ doas -u _goaccess goaccess -a --db-path "/var/db/goaccess/${domain}/" -f "/var/www/logs/${domain}/access.log" -o "/var/db/goaccess/${domain}/stats-${domain}-${date}.html"

I created a shell script, named goaccess.sh:

#!/bin/sh

###
#
# manage statistics by domain
#
##

date="$(date +'%m-%Y')"
dir_db="/var/db/goaccess"
domain="$1"

if [ -z "${domain}" ]; then 
	printf '%s %s\n' "KO" "No domain. Script stops!"
	logger "$0: no domain found as option; script stops!"
	exit 1
fi

if [ ! -d "${dir_db}/${domain}/" ]; then
	printf '%s %s\n' "KO" "The destination directory '${dir_db}/${domain}/' seems not exist!"
	logger "$0: The destination directory for goaccess not exists; script stops!"
	exit 2
fi

goaccess -a --db-path "${dir_db}/${domain}/" -f "/var/www/logs/${domain}/access.log" -o "${dir_db}/${domain}/stats-${domain}-${date}.html"

And, after setting up the _goaccess crontab:

*/15 * * * * -ns /directory/goaccess.sh domain-x.tld
0    * * * * -ns /directory/goaccess.sh domain-y.tld
0    0 * * * -ns /directory/goaccess.sh domain-z.tld

(at differents times).

No! It’s not finish…

In fact, the generated HTML file to dataviz is wrote on the database directory. The _goaccess user cant write on the web directory, and does not have access.

On the other hand, it is possible to ask the web user www to copy the HTML statistics file on the corresponding web directory.

Here, I use this following shell script, named cp_stats.sh:

#!/bin/sh

set -e
#set -x

###
#
# copy statistics files into the domain web directory
#
##

date="$(date +'%m-%Y')"
domain="$1"
dir_db="/var/db/goaccess"
dir_stats="/var/www/htdocs/${domain}/www/stats/"

if [ -z "${domain}" ]; then 
	printf '%s %s\n' "KO" "No domain. Script stops!"
	logger "$0: no domain found as option; script stops!"
	exit 1
fi

if [ ! -f "${dir_db}/${domain}/stats-${domain}-${date}.html" ]; then
	printf '%s %s\n' "KO" "The needed file '${dir_db}/${domain}/stats-${domain}-${date}.html' not exists. Script stops!"
	logger "$0: The needed file '${dir_db}/${domain}/stats-${domain}-${date}.html' not exists; script stops!"
	exit 2
fi

if [ ! -d "${dir_stats}" ]; then mkdir -p "${dir_stats}"; fi

cp "${dir_db}/${domain}/stats-${domain}-${date}.html" "${dir_stats}"
chown -R www "${dir_stats}"

You need to modify the crontab of the web user:

$ doas -u www crontab -e

Like, for instance:

*/15 * * * * -ns /directory/cp_stats.sh domain-x.tld
5    * * * * -ns /directory/cp_stats.sh domain-y.tld
5    0 * * * -ns /directory/cp_stats.sh domain-z.tld

Time-real

The real time use is done in two possible ways.

Into those contexts, we will not need the _goaccess user.

Terminal

⇒ at least, tape:

$ goaccess -f /var/www/logs/${domain}/access.log

Enjoy the aesthetic view based on monokai colors, by default.

Web proxy

Here, it’s really interesting, but a “bit complicated”:

relay httpd

I have not found solution for the httpd service. Maybe, with relayd ;-)

proxy nginx

Lets to set the server configuration to add the location directive:

location /ws {
        proxy_pass http://localhost:7890;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
}
websocket

Now, you need to execute goaccess with the www web user rights:

$ doas -u www goaccess -p /etc/goaccess/goaccess.realtime.conf -o /var/www/htdocs/huc.fr.eu.org/www/stats/realtime.html --ws-url=wss://huc.fr.eu.org:443/ws --port 7890
 [PARSING /var/www/logs/doc.huc.fr.eu.org/access.log] {0} @ {0/s}
WebSocket server ready to accept new client connections

Note: for this test, I have to:

  • create a new configuration file named goaccess.realtime.conf
  • to create a new HTML file
  • and to listen the server on the secure websocket.

Also, we can monitoring, in real time, the user, into a SSH console, with binaries like fstat, ps; see:

$ fstat -u www -n
USER     CMD          PID   FD  DEV      INUM        MODE   R/W    SZ|DV
www  goaccess   76729   wd  4,15   725760        40755    r      512
www  goaccess   76729    0  4,0     78329        20620   rw    5,0  
www  goaccess   76729    1  4,0     78329        20620   rw    5,0  
www  goaccess   76729    2  4,0     78329        20620   rw    5,0  
www  goaccess   76729    3  4,3       175        10644   rw        0
www  goaccess   76729    4  4,3       176        10644   rw        0
www  goaccess   76729    5  4,3       175        10644    w        0
www  goaccess   76729    6  4,3       176        10644    w        0
www  goaccess   76729    7 pipe 0x0 state: 
www  goaccess   76729    8 pipe 0x0 state: 
www  goaccess   76729    9* internet stream tcp 0x0 *:7890
www  goaccess   76729   10 pipe 0x0 state: 
www  goaccess   76729   11 pipe 0x0 state: 
www  goaccess   76729   12* internet stream tcp 0x0 127.0.0.1:7890 <-- 127.0.0.1:6459

$ ps aux -U www
USER       PID %CPU %MEM   VSZ   RSS TT  STAT   STARTED       TIME COMMAND
www   4071  0.0  0.1  1572  3488 ??  S      10:12AM    0:00.88 sshd: www@notty (sshd)
www  76729  0.0  0.3 10096 12752 p0  S+      7:56PM    0:01.49 goaccess -p /etc/goaccess/goaccess.realtime.conf -o /var/www/htdocs/huc.fr.eu.org/www/stats/realtime.html /var/www/logs/do

Now, go to view the realtime.html file on your web browser.

dashboard goaccess real time
dashboard goaccess real time

Note the little dash below the parameter icone, at the top left on the screen.

It seems necessary to reload the page; after some time, it lose the connector. Press F5…

Finally, we cant execute goaccess as service, either by the –daemonize option, or by configuring the dedicated configuration file.

$ doas -u www goaccess --daemonize -p /etc/goaccess/goaccess.realtime.conf -o /var/www/htdocs/huc.fr.eu.org/www/stats/realtime.html --ws-url=wss://huc.fr.eu.org:443/ws --port 7890
Daemonized GoAccess: 48646

This option only works with real time option enabled.

Egual, it’s imperative that the web user is given access to the path that will write the PID process file — otherwise, you will fail!

You need to set the pid-file option into the configuration file.

As reminder, on OpenBSD, by default, it’s on the web chroot: /var/www/run. Prefer to create a sub-directory dedicated to the user www.

Of course, all of this, it’s for the FUN, and the example! :D

Voila!

(It’s my XP… and yours‽)

Troubleshooting

Here are some errors encountered:

Permission denied

  • Couldn't open file /var/db/goaccess/xxx/I32_DATES.db: Permission denied
  • Unable to open the specified pid file. Permission denied
  • Unable to open the specified pid file. Permission denied
  1. goaccess can not write into the directory!
  2. check that directory exists.
  3. check rights user; they must match the one of the user who runs goaccess, like : _goaccess:daemon

This is the same problem during the generation of HTML files.

Example:

GoAccess - version 1.5.1 - Sep 26 2021 14:08:19
Config file: /etc/goaccess/goaccess.conf

Fatal error has occurred
Error occurred at: src/output.c - output_html - 1183
Unable to open HTML file: Permission denied.

Error opening the specified MaxMind DB file

GoAccess - version 1.5.5 - Apr  8 2022 09:03:43
Config file: /etc/goaccess/goaccess.conf

Fatal error has occurred
Error occurred at: src/geoip2.c - init_geoip - 89
Unable to open GeoIP2 database /var/db/GeoIP/GeoLite2-Country.mmdb: Error opening the specified MaxMind DB file

You have certainly activate the geoip-database option.

But did you download the necessary files and install them in the directory /var/db/GeoIP/?

No home directory

  • Make sure that the home directory for the dedicated user really exists !

Then check the existence of the path in your filesystem, without forgetting the user rights on!

  • make sure that the home directory of the dedicated user AND the dir_db variable on the goaccess.sh script matches.

Documentations

Wikipédia

Enjoy-ID! Enjoy-IT!