Massive data is becoming essential to the AI ​​ecosystem

Data storage is back in fashion, and global data Projected to reach 200 zettabytes By 2025, half of that will be in the cloud. Just five years ago, that figure was 41 zettabytes.

One of the main reasons for the rapid growth in data volumes is the shift from text-based to multimodal models and the development of hyper-intelligent and artificial intelligence systems. While many companies are vying for the AI ​​throne, Vast Data Inc., a fast-growing data computing platform, has established itself as the backbone of the data space shaped by machine learning.

The company’s outlook is bright, as the next-generation storage market is expected to continue to expand. $150 billion in 2032Vast is growing at a compound annual growth rate of 10% and has lucrative partnerships with Nvidia Corp. and cloud service providers. It’s worth asking what Vast brings to the table for data storage and how it fits into the AI ​​ecosystem.

“Data is a big problem,” Vast co-founder Jeff Denworth told theCUBE in an exclusive interview. “We’re at the center of that entire pipeline. Our system is both a distributed enterprise data warehouse and a distributed unstructured data store, so we can solve every step of the pipeline with a unified, high-performance, very affordable enterprise platform. This isn’t just training. It’s a lot more than that.”

Last year, Vast introduced its data platform. It is being marketed as a way for developers to interact at SiliconANGLE Media’s livestreaming studio, theCUBE. Data regardless of file type or locationCurrently, the company Preparing for Data Engine ImplementationAnother software layer is Coordinates events in the AI ​​pipeline the Cosmos Event It will be held in October, and TheCUBE will be bringing you exclusive coverage of the event. (*Disclosure information below)

How Vast Data Platform Manages Unstructured Data

Vast takes a non-traditional approach to data, blending data storage and database concepts to manage both unstructured and structured data in the AI ​​era, allowing users to interact with all their data on-premise and in the cloud on Vast’s distributed platform.

“During my career, I’ve seen a lot of data companies start expanding into the stack beyond their core capabilities,” said Sanjiv Mohan, industry analyst and principal at Sangimo. Vast’s Build Beyond Event“This is the first time that storage companies have come on board and started offering all these new features.”

Vast Data Platform Have Four Components: DataStore, DataSpace, DataBase, DataEngine. DataStore provides universal storage based on the respective cluster. Distributed Shared Everything ArchitectureDASE builds on Google’s concept of a shared-nothing system by separating the computing logic from the system state, which essentially allows the cluster to scale the data capacity of the platform independently of the CPU.

Eliminate East-West Traffic is one of the highlights of Vast DataBase, a transactional data warehouse that combines the capabilities of traditional databases, data warehouses, and data lakes. As an alternative to the data lake or data lakehouse architectures of Snowflake Inc. and Databricks, Inc., it focuses on unstructured data processed by AI.

“We started with the storage layer,” Denworth says. “The reason we wanted to do this was because we looked at deep learning and the power of the very large AI supercomputers that NVIDIA was trying to build and we needed a more scalable, high-performance real-time architecture. [Vast DataBase] It’s purposefully designed to take advantage of the power of flash as an exabyte-scale transactional data warehouse, and when you combine the two, you get an unlimited scale information architecture that transforms unstructured data into structured data. This is not the kind of thing that Snowflake and Databricks are really doing.”

Vast claims to have closed the observability gap that exists in data lakes and data lakehouse architectures by streaming data at any scale into its platform. Additionally, it has Vast DataSpace, which integrates multiple clouds and allows users to stream all tabular files and records into a single pipeline.

“What emerges from this is a product that is more efficient and scalable than traditional products. [Snowflake and Databricks] “It’s built for these purposes, but its capabilities are much broader because it also supports a variety of data types that don’t naturally fit into a data warehouse,” Denworth said.

Transforming the AI ​​ecosystem

The vast one Betting it all on FlashIt is powered by an architecture that makes flash more affordable and solves the input/output (IO) bottlenecks that have historically plagued the storage market, resulting in 2x to 20x improvements in pipeline performance. According to Denworth.

The final stage of simplification appears to be Vast’s DataEngine, which is based on the company’s collaboration with Nvidia. A server-level engine to orchestrate AI pipelinesIt is based on event triggers and functions: any I/O event in the DataStore can trigger a function to be executed by the DataEngine without any explicit instructions from the AI ​​developer.

“we [saw] “At a high level, we realized that the application stack was just as complex on top of all these data issues,” Denworth said. “Given that these are all data-driven challenges that our customers are facing, what we wanted to do was bring the application stack into our system and make it easier for organizations to implement enterprise AI across their entire organization and use AI tools to unlock the secrets hidden in their data.”

In March, Vast Denworth announced that it will use NVIDIA’s BlueField-3 data processing unit technology to run its large-scale data storage services, as well as a partnership with Super Micro Computer Inc., a global supplier of AI hardware solutions. Denworth has hinted at a series of announcements with key partners, including some of the biggest names in the IT industry, at its Cosmos event in October.

“Fundamentally the goal is to enable real-time, data-driven AI in every enterprise,” he said.

Building partnerships in the cloud space

Another group of Vast’s collaborators are cloud GPU providers. Growing in popularityOne reason is that it allows companies to rent expensive computer resources rather than buying them.

“What CSPs have more than anything is the power and ability to build out scalable, reliable infrastructure very quickly,” Denworth said in a previous interview. “They become the destination for next-generation enterprise workloads that can’t be easily deployed in-house by their customers.”

Vast has already announced partnerships with AI cloud providers Core Weave and Lambda LabsWe look forward to seeing how all these collaborations come to fruition and learning more about the Vast Data Engine in October.

“You will see us deliver on the promise we made last year to drive breakthrough, next-generation system architectures that solve many of our customers’ problems,” Denworth said. “As organizations begin to reshape their infrastructure agendas in a world of AI, Vast is positioned to lead. There is too much movement happening around and within Vast Data to ignore any longer.”

(* Disclosure: theCUBE is a paid media partner of Vast Data’s “Cosmos” event. Neither Vast Data Inc., the sponsor of theCUBE’s event coverage, nor any other sponsors have any editorial control over content on theCUBE or SiliconANGLE.)

Image: SiliconANGLE/Canva