From Supercomputers to LLMs: How Open Source and Ingenuity Democratize Cutting-Edge Technology
Big data, data science, machine learning, and now “open source” (more correctly called source-available, weights-available as their data and sometimes their training methods are not freely shared) Large Language Models (LLMs) are all the rage and have tons of hype, for better—and in some ways, for worse. Having spent my early career building open-source Linux supercomputing clusters for customers in Singapore and globally, the supercomputing world’s journey towards democratization mirrors what we are seeing in AI and LLMs today.
The Supercomputer Parallel: When Open Source Broke the Monopoly
Proprietary supercomputing systems like Cray that cost millions and required specialized expertise dominated the 1990s. But then came the Beowulf High Performance Computing (HPC) cluster paradigm – an open-source solution from NASA researchers Thomas Sterling and Donald Becker (whom I had the honour of hosting in Singapore and bringing him around our Sim Lim Square) who figured out how to combine regular PCs running Linux to create powerful computing clusters. Suddenly, universities and startups had access to supercomputing capabilities without breaking their budgets.
What we learned was simple – constraints breed creativity. By removing proprietary dependencies and embracing open-source design, Beowulf HPC clusters transformed supercomputing from an expensive luxury into an accessible tool. Today, these HPC clusters power everything from climate modeling, drug discovery and of course today’s AI and LLMs.
The LLM Revolution: Proprietary Walls Crumble
Fast forward to 2025. Training state-of-the-art Large Language Models (LLMs) like OpenAI’s GPT-4 and Google’s Gemini needed hundreds of millions of dollars to train in terms of GPUs and very high energy costs, proprietary datasets, and infrastructure that only Big Tech could afford. However, the open-source community would not accept this status quo.
Llama from Meta and Qwen from Alibaba – both popular “open-source” LLMs – changed everything. They are high-performing LLM models released freely for research and commercial use. However, it still required thousands of state-of-the-art GPUs costing millions of dollars to train.
Then came China’s DeepSeek, showing us that “bigger is better” isn’t always true. DeepSeek’s efficient training of its V3 and R1 models cost a fraction of what GPT-4 and Llama cost, highlighting significant energy and financial savings. Their DeepSeek models achieve performance rivaling larger models through smarter algorithms and innovative training methods.
The parallels to the Beowulf HPC Cluster revolution are clear:
- “Open source” LLMs (standing on the shoulders of others) offer ease of accessibility and often free or dramatically reduce training and inference costs
- Startups and researchers no longer depend on big cloud providers
- Organizations can customize models for specific needs without vendor lock-in
Importantly, DeepSeek releases its models under the permissive MIT open-source license, granting thousands of developers, engineers, students, and researchers immediate access to test, learn from, and deploy them.
Brute Force vs. Smarter Innovation: A Cultural Divide?
The Western approach to AI often mirrors their supercomputing strategy: scale at all costs. More parameters, more data, more GPUs – it is about overwhelming problems with sheer computational force.
In contrast, look at DeepSeek’s approach. Instead of chasing larger and larger GPU clusters, they focus on clever architecture:
- Using Mixture-of-Experts to activate only necessary parts of the model
- Prioritizing quality data over quantity
- Implementing algorithmic optimizations for faster inference
This is not just about being cost-effective – it is about finding smarter ways to solve problems under real-world constraints.
Why This Matters for Humanity’s Challenges?
Just as Beowulf HPC clusters enabled smaller labs to tackle complex problems, open-source LLMs are opening up new possibilities for engineers worldwide to solve problems that proprietary models ignore.
Think about what this means:
- A cash-strapped R&D team can build an LLM for low-resource languages which the Big Tech ignores as the market is small.
- Hospitals can train models on patient data while maintaining privacy
- Students can build grassroots led innovation cheaply and easily
When massive compute budgets no longer restrict AI, we will see innovation everywhere, driven by anyone with a problem to solve, mirroring how many current open-source software projects began.
The Future Is Open (and Hybrid)
The direction is clear – proprietary systems will continue to exist, but open source ensures they cannot stifle progress. Just as Beowulf HPC clusters did not kill supercomputers but forced them to evolve, open-source LLMs are pushing Big Tech to innovate faster and reduce costs.
What is next? I see a hybrid ecosystem emerging where proprietary and open models coexist. Imagine startups using Llama 3 or DeepSeek’s models locally for prototyping before scaling up on cloud infrastructure or adopting expensive proprietary LLM models for a specific use case.
Final Thought
The true power of AI is not confined to corporate giants or Big Tech only.
History, from supercomputing to today’s AI landscape, shows us that scarcity fuels innovation. Limited resources ignite creativity, proving that the democratization of AI is not just a possibility, but a reality in progress.
Whether it is a student with a Raspberry Pi or a developer fine-tuning models on a laptop, open-source tools are leveling the playing field. Ultimately, the future of AI is not about who has the most processing power, but who possesses the ingenuity to apply these readily available tools to real-world challenges. It is about asking “how can we make this better, faster, cheaper?”. The organizations that thrive will not be the ones with the biggest budgets, but the ones that can effectively bridge the gap between open-source AI and practical problem-solving.
Happy learning (and not necessarily on the biggest GPU cluster)!