It was first published KestraJanuary 24, 2024.
Do more with less
The technology industry in 2024 is under pressure to optimize its resources. Technology and data leaders will be required to integrate more data to support new AI-driven capabilities, while at the same time being forced to reduce costs and headcount. Even the biggest tech companies aren’t immune to this efficiency trend, judging by recent layoffs at Google, Amazon, Meta, Twitch, Spotify, and Discord.
Impact of AI on attrition and economic factors
Increased LLM capabilities are reshaping the job market, and the data space is no exception. Although it is difficult to estimate how much advances in AI are contributing to the growing wave of tech layoffs, many companies are cutting costs in existing business areas and reallocating that budget to AI development. . Dropbox cut its workforce by 16% last year and reallocated its resources to hiring AI specialists to “remain competitive.”
Economic factors such as a slowdown in VC funding and post-pandemic (late) adjustments will also influence headcount decisions.
Impact on data engineering
As organizations seek to do more with less, the demand for generalists who are proficient in cloud-native technologies, data, AI, and platform engineering is increasing. This shift has shifted the field away from highly specialized roles such as ETL and BI engineers and toward a broader emphasis on engineering skills. Data engineering teams in 2024 will look more like software engineering teams. This happens partly out of necessity, partly due to the maturity of data engineering as a field. Data teams are expected to deliver more with fewer resources, and that often requires building data products faster with smaller teams than before.
On the other hand, the quality of AI-based products depends on the quality of the underlying data, so software engineers working on AI-driven functions and data products will begin to take over many data engineering tasks such as data cleaning, validation, and governance. Masu. . Even if he adjusts the LLM based on bad data, no matter how many GPUs he throws at it, it won’t bring good results to the business. In 2024, you may find that the lines between software and data teams are blurring.
Click here to continue reading this article.
