White Papers

The Evolving Role of the Data Engineer

Issue link: https://www.qubole.com/resources/i/1243713

Contents of this Issue


Page 15 of 63

You also need to structure data so that your users don't see sen‐ sitive data to which they are not entitled. Some modern data stores provide views, as relational databases do, so that you can present to each user just the rows and columns to which they have access rights. Because data duplication is common in modern data environments, you can create custom data sets for users. Development life cycle All the trends discussed so far call for a clear set of steps in the evolution of your processes for handling data. The volume and variety of big data require repeatable, reliable processes. Secu‐ rity will be woefully lax if you deploy buggy software and pro‐ cesses, so strong software engineering practices and good orchestration tools can prevent problems. And opportunities for automation allow you to meet all these needs as well as force you to define the procedures carefully. Few organizations assign all these tasks to a single person. Data engineering is normally a team effort, with senior members of the team setting policies and other members handling designated tasks. The field of data engineering has started to get attention in the pub‐ lishing world. One blog post lists books for data engineers to read, although not all of these books focus on data engineering. Some talk about data warehousing, which is different, or about tools shared by data science and data engineering. Evaluation Process The data revolution is driving, and is driven by, a new empower‐ ment among employees at many organizational levels. This is called the democratization of data because organizations extend access to data to people below the management layers that used to have exclu‐ sive access. The field of BI is being transformed, like so many other organizational practices. As a data engineer, you need to work closely with the potential users of your data to give them the data sets they need, in the structure they need. Questions to ask include: What data sets do users need? This helps you prioritize which to process and ensure that they can be found in catalogs. 8 | The Evolving Role of the Data Engineer

Articles in this issue

Links on this page

view archives of White Papers - The Evolving Role of the Data Engineer