White Papers

The Evolving Role of the Data Engineer

Issue link: https://www.qubole.com/resources/i/1243713

Contents of this Issue


Page 51 of 63

Metrics and Evaluation Data analysis is driving modern organizational decision making, and it can make your work more effective, too. Collecting metrics is easy—all too easy. Cloud vendors practically urge their metrics on you, and metrics are prolifically available on- premises as well. The first and hardest job is determining what met‐ rics really offer the information that can help your organization run better; you will probably change the metrics you collect as you learn more about the organization's needs. Cloud vendors live by their metrics, so naturally they offer metrics to clients on every cloud service. AWS integrates all these metrics through CloudWatch, Azure through Azure Monitor, and GCP through a metrics reporting service. Best Practices • You can automate the collection of the metrics you choose to monitor and view them on dash‐ boards. • For on-premises deployments, you can collect metrics using Ganglia and produce graphs or dashboards through a variety of tools. Commer‐ cial tools are also available. Business metrics It's nearly impossible to make a link between available metrics and big, mission-driven questions such as "Can our employees make decisions that were not open to them before?" or "Did analytics help us improve our annual results?" But organizations can measure some aspects of user behavior, such as: • How quickly a business user receives the data they request. If this is unsatisfactory, you might need to restructure your data, create new data sets and tables of aggregate data, or add hard‐ ware and software resources to speed up transfers. • How often each data set is being used. A rarely used data set is a waste of money. You have to determine whether it has a purpose within your organization. If it could potentially provide value to 44 | The Evolving Role of the Data Engineer

Articles in this issue

Links on this page

view archives of White Papers - The Evolving Role of the Data Engineer