With the advent of technology, others can easily catch up (knowledge spillover). Before AlphaGo came out, professional Go players could do a nine-stone handicap with computer. After AlphaGo, there were open source projects, and now even your smartphone can beat professional Go players.

xAI’s Grok, when asked certain questions, replies ‘According to OpenAI’s policy, I cannot answer this question.’ Google’s Gemini also has moments where it behaves as if it’s GPT.

Although OpenAI’s terms state that their API cannot be used as a model for training competitors, in practice, it’s virtually impossible to enforce. As soon as someone publishes an article written by GPT on the internet, other companies scraping web data will inevitably capture it.

So far, no model has significantly surpassed OpenAI’s, but it’s not really a case of overtaking. Technology naturally evolves from what already exists. It’s just that the number of competitors has suddenly increased, and those following can replicate similar achievements at a lower cost.

Meta’s LLAMA 2 has a mobile version for local use. Google’s Gemini also has a mobile local version.

The fact that these technologies have good outcome performance even on mobile devices shows that often the limitation is not computing power, but knowledge. John Carmack, a prominent figure in the game engine industry, once said that too much research focuses on stacking hardware rather than on algorithms.

However, looking at AI development, to run on a mobile device, the premise is that it must first be developed on a supercomputer and then compressed for mobile use.

A tsunami is already happening, and I find it terrifying that most people are not yet aware of the era of AI. On Reddit, when I discuss using AI for certain tasks, I often get downvoted. I think a big reason for this is that past intuitions are no longer applicable. It took eight years from the appearance of AlphaGo to ChatGPT, and people think that it will take another eight years for a technology of GPT’s magnitude to emerge. But AI is now growing exponentially. AI chips are developed by AI, and AI algorithms are developed by AI, accelerating themselves.

Last year, everyone was joking that GPT could only make random, nonsensical statements and couldn’t think long-term. This year, OpenAI has Q*, and Google has AlphaCode2, both of which have long-term thinking capabilities.

The difference between knowledge and memory is that memory is a fixed retrieval, like a Google search for existing web pages. Knowledge is deduction, like AIGC drawing pictures or writing articles. It’s about inferring what color a pixel should be or what word should come next.

I’ve been studying learning theories lately. From an evolutionary perspective, the human brain is structured for knowledge learning. School is painful because it’s all about memorizing, even memorizing the process of solving problems, instead of learning a piece of knowledge and then deducing from it.

The principle of knowledge learning is model compression. After seeing many scenarios, one finds a method that can explain them (a compressed model) and then applies it to new, unseen problems. The human brain, being knowledge-driven, is adept at abstract conceptual reasoning to adapt to new environments.

For the next generation’s education, perhaps we can encourage students to get used to encountering a wide range of content and to attempt compressive thinking. How to apply the fewest concepts to the most scenarios, and when encountering something new, to update the compressed model. I recall 高川格, a Go YouTube channel, where the creator initially found AI exciting. But after AI dominated human players, he struggled to find meaning in Go. After years of searching, his interpretation of Go became:

‘In the past, only what professional Go players said was the truth. Amateurs playing differently from professionals would be criticized. Humans can no longer defeat AI, and there’s no need to pursue the truth anymore. Also, different AIs may support the same move on the board. But why play the game this way? You need to provide a reason, not an AI’s reason, but your own.’

Model compression in learning theory is about giving ‘a reason’ after seeing many things. Modern art has also evolved in this direction, from the Renaissance, classical, academic, abstract… to now where anything can be art. Performance art is also art. Anything that doesn’t fit into a category is just called modern art.

The point for an artist is to interpret, to ‘give a reason.’ Some art schools even base graduation on how students interpret their work; the work itself is secondary.

A TED Talk by NiceChord(Wiwi) mentioned that in the classical period, because there were no phonographs, many performers were needed. But now, with iPods, everyone can hear music close to performance quality. Do we still need to produce so many human iPods? We should shift the focus more towards creation.”

help me naming this post title in short