How are Indian firms training LLMs?, Pg10

Key Highlights:

• Bengaluru-based startup Sarvam AI launched two Large Language Models (LLMs) trained on 35 billion and 105 billion parameters.
• The IndiaAI mission is subsidizing domestic training efforts by commissioning over 36,000 GPUs in data centers operated by Indian firms.
• The government provided Sarvam AI access to 4,096 GPUs from its common compute cluster, with a subsidy estimated at almost ₹100 crore.
• Mixture of Experts (MoE) architecture was a key breakthrough, activating only a fraction of parameters, making queries faster and consuming fewer resources.

Detailed Insights:

• Training LLMs requires clusters of Graphics Processing Units (GPUs), costing millions of dollars in hardware and electricity.
• Indian languages are underrepresented in internet data, creating challenges for training LLMs to perform well in these languages.
• The Ministry of Electronics and Information Technology (MeitY) encourages domestic LLM development, believing foreign LLMs may not prioritize Indian languages.
• Sarvam AI's models aim for accuracy, usefulness, efficiency, and alignment for the Indian context before training larger foundational models.
• BharatGen, an IIT Bombay-incubated firm, trained a multilingual 17 billion parameter model for use in sectors like education and healthcare.
• Making the LLM open source would allow outside experts to scrutinize the claims the firm has made about the model.

Key Concepts Involved:

• Large Language Models (LLMs): AI systems trained on vast amounts of data to generate human-like text.
• Parameters: Variables that an AI model learns during training to make predictions or generate outputs.
• IndiaAI Mission: Government initiative to promote AI development and adoption in India through infrastructure and subsidies.
• Mixture of Experts (MoE): An AI architecture that activates only a subset of parameters during inference, improving efficiency.