At Kudi, we believe that people of all classes should have access to financial services and seamlessly perform transactions. We make this possible through our agency banking network.
Logging is an important part of application development. It helps provide visibility into what is going on inside your code execution environment. However, it's quite easy to treat logging like something that is done only to meet compliance requirements. By so doing, it becomes easy to even forget that logs exist.
Google Cloud Logging provides realtime log management and analysis at scale. It is secure, scalable, and fully managed, meaning you don't need to think about configuring infrastructure for storage or processing, and you never need to worry that the infrastructure might not be sufficient when you need it.
We had an issue with a database driver that led to some important application data not being stored.
The discovery of this issue led to a moment of panic!
By inspecting the code, we found that the data being sent to the database was being logged via Google Cloud Logging.
Google Cloud Logging provides Logs Explorer, an interface for analyzing logs data. It lets users search, sort, and analyze logs through flexible query statements along with rich histogram visualizations, simple field explorers, and provides the ability to save the queries.
By utilizing the gcloud command line tool, we were able to read the logs and export them into a text file. This command produces unstructured data.
At this point we could have processed the data using any tool of our choice. In our case, we chose to upload the file into Google Cloud Storage and from their we imported it into BigQuery and used the REGEXP_EXTRACT string function to parse out the components we were interested in.
We will write a separate article at a future time about this stage of the process, but suffice it to say that we use BigQuery as a tool belt.
If you find our use of technology to be of interest and would like to join our team, please take a look at our careers page here, or drop us a note at engineering@kudi.com. Also, please subscribe to receive updates from us on new posts and job openings.
Comments