Ingesting and parsing custom logs using Google Cloud’s Ops Agent_

Google Cloud’s Ops Agent, as part of Cloud Operations, is in charge of collecting logs and metrics from your Compute Engine instances. It uses Fluent Bit for logs and the OpenTelemetry Collector for metrics. In this article, we will deep dive on the logging piece, focusing on how to parse logs using regular expressions.

So, why would you want to parse logs at the instance level instead of doing the parsing and manipulation on Logs Explorer? Well, there are specific use cases where you may not want certain information on the logs to leave the VM — for example, logs with confidential information. In such cases, with Ops Agent, we are able to parse logs and send only the pieces of information we want to Cloud Operations.

For the purpose of this tutorial, we will suppose we would like to monitor our application called my-app. For this, we will configure Ops Agent to push the logfile /var/log/my-app.log to Logs Explorer. However, we don’t want to push the entire content of the log since it has some PII data. Our example logfile looks like this:

Of course, for the given scenario, you could also use Google Cloud’s Cloud Data Loss Prevention to discover and protect PII data. It’s a very powerful serverless service that integrates seamlessly with other Google Cloud products. Also, it provides features such as data masking, de-identification mechanisms and more. Consider using DLP for big amounts of data and if they reside in different sources other that Compute Engine VMs, such as BigQuery, Cloud Storage or Datastore. Also, if you need more complex data manipulation such as obfuscation, masking and classification.

Leave a Comment