Google Cloud Skilled Companies additionally gives assets that may allow you to automate the deployment of the GCP instruments concerned on this resolution. Even higher, the providers used are fully-managed: no work is required post-deployment.
Is that this resolution proper for me?
Earlier than continuing, let’s resolve if the instruments on this publish are proper to your group. Cloud Identification Premium has a function that permits you to export Cloud Identity logs straight to BigQuery. This can be ample in case your group solely wants to research the logs in BigQuery. Nevertheless, it’s possible you’ll wish to export the logs to Cloud Logging for retention or additional processing as a part of your regular logging processes.
GCP additionally has a G Suite audit logging function which robotically publishes some Cloud Identification logs into Cloud Logging. You’ll be able to discover which Cloud Identification logs this function covers within the documentation. The G Suite log exporter software we are going to discover on this publish gives extra protection for getting Cellular, OAuth Token, and Drive logs into Cloud Logging, and likewise permits the consumer to specify precisely which logs they wish to ingest from Cloud Identification.
If both of those conditions are related to your group, maintain studying!
The instruments we use
The G Suite log exporter is an open-source software developed and maintained by Google Cloud Skilled Companies. It handles exporting information from Cloud Identification by calling G Suite’s Reports API. It specifies Cloud Logging on GCP because the vacation spot to your logs, grabs the Cloud Identification logs, does some cleanup and reformatting, and writes to Cloud Logging utilizing the Cloud Logging API.
One strategy to run this software is to spin up a digital machine utilizing Google Compute Engine. You can import and execute the software as a Python package deal and arrange a cronjob that runs the software on a cadence. We even present a Terraform module that may automate this setup for you. It appears easy sufficient, however there are some issues you will need to contemplate in the event you take this path, together with the right way to safe your VM and what mission and VPC it belongs to.
Another method is to make use of Google-managed providers to execute this code. Cloud Functions provides you a serverless platform for event-based code execution—no must spin up or handle any assets to run the code. Cloud Scheduler is Google’s absolutely managed enterprise-grade cronjob scheduler. You’ll be able to combine a Cloud Operate with a Cloud Scheduler job in order that your code executes robotically on a schedule, per the next steps:
Create a Cloud Operate that subscribes to a Cloud Pub/Sub matter
Create a Pub/Sub matter to set off that operate
Create a Cloud Scheduler job that invokes the Pub/Sub set off
Run the Cloud Scheduler job.
We additionally present open-source examples that may allow you to take this method, utilizing a script or a Terraform module. Publish-deployment, the Cloud Operate can be triggered by the recurring Cloud Scheduler job, and the GSuite log exporter software will execute indefinitely. That’s it! You now have up-to-date Cloud Identification logs in Cloud Logging. And since we’re utilizing fully-managed GCP providers, there’s no additional effort required.
Customizing the answer
The open-source examples above can be personalized to suit your wants. Let’s check out the one which makes use of a script.
On this instance, the default deploy.sh script creates a Cloud Scheduler job that triggers the exporter software each 15 minutes. However, let’s say your group wants to drag logs each 5 minutes to fulfill safety necessities. You’ll be able to merely change the “–schedule” flag on this file in order that the exporter software is fired as usually as you’d like. The cadence is outlined in unix-cron format.