Electronic Enthusiast Network reports (by Huang Jingjing): Despite the current AI training primarily employing the GPU+HBM solution, some new technologies still hope to further break through the bottleneck issues brought about by data transfer in storage. Recently, NEO Semiconductor announced the development of its 3D X-AI chip technology, aiming to replace the DRAM chips in the current high-bandwidth memory (HBM), by implementing AI processing in 3D DRAM to solve the data bus problem.

Typically, the current AI chip architecture stores data in high-bandwidth memory and transfers the data to the GPU via a data bus to execute AI algorithms (mathematical computations). This architecture is inefficient, with the data bus causing long delays and high power consumption.

The 3D X-AI, however, uses storage units to simulate synapses in neural networks. It supports data storage and artificial intelligence operations on the same chip. Data stored in the storage units is directly used to generate the output of the neural network without any mathematical computations, thereby greatly enhancing AI performance and significantly saving energy.

Advertisement

The 3D X-AI chip is a 3D DRAM with AI processing capabilities on the same chip; it has a neuron circuit layer at the bottom and 300 storage unit layers at the top, with a capacity of 128 GB. This innovative chip can increase the performance of AI chips by 100 times and reduce power consumption by 99%. It has 8 times higher density, making it very suitable for storing large language models (LLM) used in generative artificial intelligence applications, such as Chat GPT, Gemini, and CoPilot.

NEO Semiconductor introduces that AI chips using NEO's 3D X-AI technology can achieve a 100X performance acceleration: including 8000 neuron circuits that can perform AI processing in 3D memory; a 99% reduction in power consumption: minimizing the need to transfer data to the GPU for computation, thereby reducing the power consumption and heat generated by the data bus; 8 times memory density: including 300 memory layers, allowing for the storage of larger AI models. According to NEO's estimates, each chip can support up to 10 TB/s of AI processing throughput. Stacking 12 3D X-AI chips in an HBM package can achieve a 120 TB/s processing throughput, a 100X performance improvement.

The 3D DRAM is also one of the research and development directions of NEO Semiconductor. Unlike traditional DRAM with horizontally placed storage units, 3D DRAM stacks storage units vertically, significantly increasing the storage capacity per unit area and improving efficiency, becoming a key development direction for the next generation of DRAM.

NEO states that dynamic random-access memory (DRAM) is used to support processors, making DRAM more widely used in electronic devices. However, the speed of processor growth outpaces the speed of memory across multiple generations, resulting in an expanding "performance gap" year by year. Power-sensitive environments like cloud data centers increasingly rely on higher-powered processors to meet performance requirements, but this reduces the power available for memory.

Adopting the X-DRAM architecture can reduce power consumption, lower latency, and increase throughput to overcome these and other challenges encountered when using traditional DRAM. This provides commercial systems (such as servers) with higher performance, mobile devices (such as smartphones) with longer battery life, edge computing devices (such as routers) with more capabilities, and IoT objects (such as gateways) with new deployment options.The unit array structure of 3D X-DRAM is similar to that of 3D NAND Flash, utilizing FBC (Floating Body Cell) technology, which can form a vertical structure by adding layer masks, thereby achieving high yield, low cost, and a significant increase in density. NEO states that the 3D X-DRAM technology can produce a 128Gbit DRAM chip with 230 layers, which is eight times the density of current DRAM. In recent years, memory manufacturers such as SK Hynix, Samsung Electronics, and Micron have been researching and developing 3D DRAM technology to meet the demand for high-performance, high-capacity memory under the AI wave.