White Papers

The Evolving Role of the Data Engineer

Issue link: https://www.qubole.com/resources/i/1243713

Contents of this Issue


Page 8 of 63

The Evolving Role of the Data Engineer Every organization can benefit from data, used adeptly in coordina‐ tion with the organization's goals. Today's process of accumulating enormous amounts of information from different sources—big data in common parlance—is like a powerful telescope that lets you see further into the universe and learn about trends you couldn't track before. The term data engineer is relatively new, and the role appears only sporadically in technical literature. It is the data scientists who are exalted as today's heroes: the high-flying fighter pilots of big data who create machine learning models for predictions and other ana‐ lytics. But no airplane could fly without a small army of mechanics and other trained staff to prepare the planes. In this analogy, the data engineer is like the airplane mechanic, preparing the data that enables the data scientist to carry out organizational goals. When you add up all the tasks that fit under this role—getting the data, cleaning it, creating enhanced versions—observers often claim that data engineering comprises 80 to 90% of the work organizations do with data. Why did data engineering become so important? During the 1990s, the world underwent a momentous transformation that reached more and more deeply into our lives and brought cascading impacts upon just about every career. The transformation was driven by a hodgepodge of trends: 1

Articles in this issue

view archives of White Papers - The Evolving Role of the Data Engineer