Integrating Analytics for WSO2 API Manager 4.0.0 with Fluent Bit, Grafana Loki, Grafana Stack

5 min readNov 7, 2021

Analytics is one of the feature that we require when we are implementing a monitoring solution. Same applies to WSO2 deployments also. When it comes to WSO2 Analytics it provides different set of solutions for example there is a APIM Analytics Server which we can integrate along with API Manager and also there is another latest solution is to point the WSO2 APIM to Choreo Analytics. But in some cases we require not to push our analytics data to a third party system or cloud solution, due to that there is a need to integrate the current WSO2 APIM to a external system to provide us the Usage Analytics Statistics.

Here, my intention is to explain about how we can integrate the Fluent Bit, Grafana Loki and Grafana Dashboard to achieve the task.

Below flow diagram illustrate what we are going to implement and how it interacts with each other.

1. Building the Publisher Agent

Go to https://github.com/ajanthanblog/log-events and clone the repository and build the log-events.

Copy the built Jar to <APIM_HOME>/repository/components/lib.

Go to <APIM_HOME>/repository/conf/deployment.toml and add the below, remove the default one already in the file and add this as new.

To enable the log, add the below in log4j.properties file.

Also do not forgot to update the reporter module to the logger as well.

Now start the APIM and create a API for our testing purpose.

2. Setting up Grafana Loki

Execute the below commands to setup the initial Grafana loki distribution.

now we need to set a directory for storing the data.

Create a Loki Configuration File under:

and add the content as below:

To run it as a service in Ubuntu use the below command and execute.

Then execute the below commands to reload the configuration and start the service.

Once properly stated we can view a log like below:

3. Setting up FluentBit

There are two options to set this up.

We can install as a package for Ubuntu using the link https://docs.fluentbit.io/manual/installation/linux/ubuntu, this will be comes as td-agent-bit, so the commands can be different.
For my use case here I’m using the Source Code distribution from the https://fluentbit.io/releases/1.7/fluent-bit-1.7.4.tar.gz which was recommended to fetch the source based on the link https://docs.fluentbit.io/manual/installation/sources/download-source-code
Note if you are using sudo snap install fluent-bit, then you may encounter permission issues in executing the config files as I encountered. Please take a note on this.

Once extracted go to home directory and execute the below commands. Some dependencies also needed as below.

If got any error like “No CMAKE_CXX_COMPILER could be found.” this, execute the below command:

Verify the Setup Working Properly:

4. Configuring Fluent Bit to pull and push the carbon logs to Grafan Loki

To setup Fluent Bit client for Loki first we need to build the plugin. Go to

https://github.com/grafana/loki.git and clone the repository. Here I’m using the v1.6.1, if you are using the latest versions there can be mismatch in the folder structure as it has been updated in latest versions.

Go to loki home folder and execute:

make fluent-bit-plugin

The generated plugin will be under the name out_loki.so.

Now we need to create a config file to pull the logs and push it to Loki using Fluent Bit.

Under /home/ajanthan/blogs/fluent-bit create two files Create a file called loki-parsers.conf and fluent-bit.conf

loki-parsers.conf

Note: here the important part is the regex filter, use the To verify the fluent bit regex we can use the https://rubular.com/ online tool. Also this regex can be optimized and I’m using this here for this demonstration.

fluent-bit.conf

And execute the below command to start the Fluent Bit pull and push. To execute this command go to the Fluent Bit build directory

in my case it is:

and the execute:

once you successfully started can view a log like below:

5. Setting up Grafana Dashboard

Go to http://localhost:3000/ and enter admin/admin default username and password and you will be able to login.

now we need to add Loki to the data source then we can use the data from it for visualizing in the dashboard.

Go to Data Sources and add it as below screen:

After that go to Manage Dashboards and Create the dashboards as needed, here my intention is to create a dashboard by adding a API Name get the Total Request Count and show the logs based on the range we have selected.

The dashboard will look like below:

The configurations for each configured as below:

API Name: This is a Variable, click the settings icon and you can view it as below:

All Request Count:

Logs:

That’s it.

Note: This is a initial step in integrating the Grafana Dashboard. The dashboard can be modified based on your need with additional components and also can be done with further rectifying the Queries used in Grafana Loki.

Specific Things to build for future implementation: