The Firehose is out, Syslog is In | Part 2

The Firehose is out, Syslog is In | Part 2

Matthew Kocher and Nick Kuhn wrote this blog.

Matthew and I introduced Aggregrate Syslog drains in our initial blog in March: The Firehose is Out, Syslog is In | Part 1. Now we dive into configuring Aggregate Syslog drains on your platform and how you can easily integrate them into popular logging platforms.

Configuring Tanzu Application Service to use Aggregate Drains:

Configuring the platform using the Aggregate Syslog drain feature is a simple step performed inside Tanzu Operations Manager. You will need to navigate to the “System Logging” tab within the TAS tile.

Look for the “Aggregate log and metric drain destination” input box.

You will need to configure a string with the following information:

• SYSLOG_ENDPOINT: The SYSLOG endpoint to send logs. This can be a VM, load-balancer entry, or any valid endpoint that accepts Syslog traffic.
• SYSLOG_PORT: The port to be used for your syslog endpoint. Port 514 is a typical endpoint Non-TLS-based Syslog, and TLS-based Syslog is 6514. This value can be any port depending on the solution being used.
• ?include-metrics-deprecated=true&disable-metadata=false – are the entries to allow metrics to flow through the Aggregrate Syslog drains and apply certain labels to make it more useful for logging systems. We are considering making these more user-friendly in future releases. 

Sample String: syslog://{SYSLOG_ENDPOINT:SYLOG_PORT?include-metrics-deprecated=true&disable-metadata=false

Populated Example String: syslog://aria-logs.kuhn-labs.com:514?include-metrics-deprecated=true&disable-metadata=false

In this example, “aria-logs.kuhn-labs.com” is the Cluster DNS entry for our logging platform, and we will send logs over port 514.

If you want to set up custom field extraction for these values, select the application name from a log entry, right-click, and then click extract field.

Once this field is configured, you will want to run an “Apply Changes” for it to take effect.

Platform Logging with Aria Operations for Logs

In the example above, I configured my Syslog target to go to my Aria Operations for Logs cluster. In a past life, I used Aria Ops for Logs (formerly known as Log Insight) to archive logs from dozens of Cloud Foundry environments using a custom firehose nozzle. This required running multiple instances of this nozzle per foundation to ingest all of the logs egressing the platform. This was sometimes problematic as maintaining all of these nozzle endpoints used a considerable amount of compute power, and these applications had to be patched, restarted, and updated to use new features released on Tanzu Application Service. But there is some excellent news to share! These nozzles are no longer required – as the Aggregate Syslog function replaces all of this functionality. So a platform engineer does not need to worry about monitoring the health and number of the nozzles running on the platform.

If we dive into Aria Logs after the “Apply Changes” has been completed and do a simple search for “VMware Tanzu Application Service,” we can see many logs with “product=VMware Tanzu Application Service” following into the system.

If we take this one step further and add a text search for an application name, say “spring-music-homelab,” we should find logs from this application running on my foundation.

If you want to set up custom field extraction for these values, select the application name from a log entry, right-click, and then click extract field.

You can adjust the values used in the field extraction to meet your needs, but in this example, I created a custom field (cf-app-name) to extract all the cloud foundry application names as extracted values. This will make looking for logs for a specific application very easy. 

Now this is just scratching the surface of using Aria Ops for Logs and Tanzu Application Service together. Still, it does show the power of how it is easy to set up and start consuming platform logs using the Aggregate Syslog Drain feature. You may already be licensed for Aria Ops for Logs in your existing contract with VMware, so it may be a very interesting way of driving cost savings with your current logging provider.

Platform Logging with Splunk

Many users of TAS archive all of the logs and metrics coming from Cloud Foundry to Splunk using the Splunk Firehose Nozzle. By combining an aggregate drain and the (officially supported!) Splunk Connect For Syslog open-source project, we can create a very similar experience to using the Splunk nozzle.

First, we created a osnix index in Splunk, as it is the default destination for Syslog messages coming from SC4S. Then we created a Splunk HEC token. For our testing, we gave it access to all indexes. The best practice, of course, would be to provide it with access only to the indexes it needs.

We then followed the installation instructions for SC4S on Ubuntu. We added our HEC URL and Token to the env_file and restarted SC4S to ensure it had taken effect.

Now, you can search the main index in Splunk and see that syslog-ng inside SC4S is starting up. You can also run `netstat -lntp | grep syslog-ng` on the SC4S vm and see it listening on port 601. It’s worth noting that it will also be listening on 514. Port 601 is configured to accept messages with octet counting framing, the Syslog format used by Cloud Foundry’s Syslog Agent.

We then log into Ops Manager and configure the aggregate drain as depicted above, `syslog://10.0.4.24:601?include-metrics-deprecated=true&disable-metadata=false`.

Once you’ve done all this, you’ll see the messages nicely flowing to Splunk:

Here you see a flagship TAS-3.0 feature, Container Log Rate metrics, in Splunk. Splunk allows querying on the nested keys using dot notation. The format is not a drop-in replacement for the Splunk Nozzle, so you may need to update any saved searches, dashboards, or alerts.

Save time and money with Aggregate Syslog Drains!

No matter your logging platform of choice, as long as it accepts Syslog, you should be able to use the new Aggregate Syslog Drain feature. This should save platform engineers time and money by no longer having to run, maintain, and worry about their nozzle deployments. 

If you try this out, please let us know on the Cloud Foundry Slack in the #logging-and-metrics channel. We’d love to hear how it’s working for you.