How data engineers should prepare for the world of AI

There’s been a lot of talk lately about how the AI ​​revolution will reduce the role of data engineers. I believe not. In fact, data expertise will be more important than ever. However, data professionals will need to acquire new skills in order for their organizations to make the most of his AI and improve their future career prospects.

AI presents an opportunity for organizations to derive more value from their data and do so more efficiently, but it cannot do this alone. Data engineers need to learn where and how to apply technology, and which models and tools to use in which situations.

Here are four areas where AI will transform data analytics next year and the skills data engineers need to acquire to meet these needs.

Build smarter data pipelines

Data pipelines combine raw data, unstructured data, and unorganized data sources. The engineer’s job is to extract intelligence from those sources and provide valuable insights. AI is about to transform that work.

Inserting AI into data pipelines greatly accelerates the ability of data engineers to extract value and insights. For example, imagine a company has a database of customer service records and other text documents. With a few lines of SQL, an engineer can connect her AI model to a pipeline and tell it to extract rich insights from a text file. Doing this manually can take hours, and some of the most valuable insights can only be discovered by her AI.

Data engineers who understand where and how to apply AI models to get the most value from their data pipelines are invaluable to organizations, but they need to know which models to choose and how to do so. Requires new skills to apply.

Reduce data mapping and enhance your data strategy

Different data sources often store information differently. For example, one source system might refer to the state name as “Massachusetts,” while another source system might use the abbreviation “MA.”

Mapping data to ensure data consistency and non-duplication is a job tailor-made for AI. An engineer can create a prompt that essentially says, “Take these 20 customer data sources and build a legitimate customer database,” and the AI ​​will complete the task in significantly less time.

It requires knowledge of how to create good prompts, but more importantly, it frees up engineers’ time so they can spend less time mapping data and more time focusing on their organization’s data strategy and data architecture. You will be able to spend more time.

The ultimate goal is to understand all the data sources available to your organization and how to best utilize them to achieve your business goals. By handing off tasks like data mapping to AI models, you can free up time for higher-level work.

BI analysts need to up their game

Business intelligence (BI) analysts currently spend much of their time creating static reports for business leaders. If these readers ask additional questions about the data, analysts must run new queries to generate supplementary reports. Generative AI will dramatically change these executive expectations.

As executives gain experience with AI-driven chatbots, they will expect to interact with business reports in a similar conversational manner. This requires BI analysts to step up their game and learn how to provide interactive capabilities. Rather than creating static graphs, you need to understand the pipelines, plugins, and prompts required to create dynamic, interactive reports.

Cloud data platforms incorporate some of these capabilities in a low-code manner, giving BI analysts the opportunity to expand their skills to address new requirements. But there’s a learning curve, and mastering those skills will be her challenge in 2024.

Manage third-party AI services

When the cloud became popular 10 years ago, IT teams spent less time building infrastructure and software and more time managing third-party cloud services. Data scientists are about to experience a similar transition.

Growth in Gen AI will require data scientists to increasingly collaborate with external vendors that provide AI models, datasets, and other services. Being familiar with options, choosing the right model for the task at hand, and managing relationships with third parties are important skills to master.

I’m looking forward to lots of fun things in the future.

Today, many data teams say they are stuck in a reactive mode, constantly responding to the latest job requests or fixing broken applications. That’s not fun for everyone, but the influx of AI into data engineering will change that.

AI allows engineers to automate the most tedious parts of their jobs, freeing up time to think about the big picture. Although this will require new skills, it will allow us to focus on more strategic and proactive work, making our engineers even more valuable to our teams and making our jobs more enjoyable.

Jeff Hollan is Director of Product Management at Snowflake.