Sakana claims its AI-generated paper passed peer review a" but it's a bit more nuanced than that

Published in Science and Technology on Saturday, August 2nd 2025 at 23:37 GMT by TechCrunch
^{🞛 This publication is a summary or evaluation of another publication 🞛 This publication contains editorial commentary or bias from the source}

Sakana said its AI generated the first peer-reviewed scientific publication. But while the claim isn't untrue, there are caveats to note.

Sakana AI's Bold Claim: An AI-Generated Paper 'Passes' Peer Review, But the Reality is Far More Complex

In the rapidly evolving world of artificial intelligence, where breakthroughs seem to emerge daily, a recent announcement from Tokyo-based startup Sakana AI has sparked both excitement and skepticism. The company proclaimed that a scientific paper entirely generated by its AI model had successfully passed the rigorous peer-review process for a prestigious machine learning conference. This claim, if fully accurate, would mark a watershed moment, suggesting that AI could soon automate not just routine tasks but the very core of scientific discovery and validation. However, a closer examination reveals a more nuanced story—one that highlights the blurred lines between human ingenuity and machine assistance, the intricacies of academic review processes, and the potential pitfalls of hype in the AI industry.

Sakana AI, founded in 2023 by former Google Brain researchers David Ha and Llion Jones, has quickly positioned itself as a key player in Japan's burgeoning AI ecosystem. The startup focuses on developing efficient, scalable AI models inspired by natural processes, such as evolutionary algorithms and collective intelligence. Their name, "Sakana," meaning "fish" in Japanese, evokes the idea of AI systems that mimic schooling behaviors to solve complex problems collaboratively. Backed by significant venture capital, including investments from global firms, Sakana has been pushing boundaries in areas like automated model discovery and AI-driven research acceleration. This latest endeavor fits into their broader mission: to create AI that can "evolve" and generate novel ideas without heavy human intervention.

At the heart of the controversy is a paper titled "EvoJax: Evolutionary Optimization in JAX for Protein Design," which Sakana submitted to the International Conference on Learning Representations (ICLR) 2025. According to Sakana's blog post and public statements, the paper was produced using their proprietary AI system, which they describe as an "AI Scientist." This system reportedly handled everything from ideation and literature review to code implementation, experimentation, and even drafting the manuscript. The company emphasized that the AI operated with minimal human oversight, essentially conducting the research autonomously. They claimed this AI-generated work not only produced meaningful scientific contributions but also cleared the peer-review hurdle, implying a level of quality indistinguishable from human-authored papers.

The excitement stemmed from the implications: if AI can generate peer-reviewed research, it could democratize science, accelerate discoveries in fields like drug design and climate modeling, and address the global shortage of researchers. Sakana's founders drew parallels to historical milestones, likening it to the advent of calculators in mathematics or AlphaGo's defeat of human Go champions. Social media buzzed with reactions, from awe at the potential for AI to "do science" to concerns about job displacement for academics and the erosion of human creativity.

Yet, as details emerged, the narrative began to unravel—or at least complicate. The paper wasn't accepted into the main ICLR conference track, which is known for its stringent, double-blind peer review involving multiple expert evaluators. Instead, it was accepted into a workshop affiliated with ICLR, specifically the "Machine Learning for Genomics" workshop. Workshops at such conferences often have lighter review processes, sometimes single-blind or even non-anonymous, and they serve as forums for preliminary or exploratory work rather than fully vetted contributions. In this case, the review for the workshop involved a simpler assessment, not the full gauntlet of scrutiny that main-track papers endure. Sakana's claim of "passing peer review" thus requires asterisks; it's accurate in a technical sense but overlooks the distinction between workshop and conference acceptance, which carries different weights in academia.

Moreover, the extent of AI involvement isn't as absolute as initially portrayed. Digging into Sakana's own disclosures, it's clear that while the AI handled significant portions—such as generating hypotheses, writing code in the JAX framework for evolutionary optimization, and running simulations on protein design tasks—human researchers were deeply involved. The founders admitted to providing initial prompts, curating datasets, and editing the final manuscript for clarity and coherence. In fact, the paper lists human authors, including Ha and Jones, alongside the AI system credited as a co-author in a symbolic gesture. This hybrid approach aligns with emerging practices in AI-assisted research, where tools like large language models (LLMs) augment human efforts rather than replace them entirely. For instance, the AI used techniques from evolutionary computation to optimize protein structures, building on existing libraries like EvoJax, but humans verified the results and ensured scientific validity.

This nuance echoes broader debates in the AI community about what constitutes "AI-generated" content. Similar experiments have made headlines before, such as when researchers used GPT models to draft papers or when OpenAI's systems assisted in theorem proving. However, outright claims of fully autonomous AI science remain rare and often contested. Critics point out that current AI lacks true understanding or creativity; it excels at pattern matching and recombination but struggles with novel insights or handling edge cases without human guidance. In Sakana's case, the protein design focus is particularly apt, as evolutionary algorithms have long been used in bioinformatics, making it a domain where AI can shine by iterating on established methods.

The peer-review process itself adds layers of complexity. ICLR, one of the top venues for machine learning research, employs a system where submissions are evaluated on criteria like novelty, technical soundness, and impact. Reviewers, typically anonymous experts, score papers and provide feedback, with acceptance rates hovering around 30% for the main track. For workshops, the bar is lower, often prioritizing discussion-worthy ideas over polished perfection. Sakana's paper received positive feedback, with reviewers noting its innovative use of JAX for scalable optimization and potential applications in synthetic biology. However, some comments highlighted limitations, such as the need for more benchmarking against state-of-the-art methods or deeper analysis of failure modes—issues that might have doomed it in a stricter review.

Experts in the field have weighed in, offering a spectrum of views. Some, like AI ethicist Timnit Gebru, caution against overhyping such achievements, arguing that they can mislead the public about AI's capabilities and distract from pressing issues like bias in training data or environmental costs of model training. Others, including machine learning pioneer Yoshua Bengio, see promise in AI-augmented science but stress the importance of transparency. In interviews, Sakana's team has defended their approach, stating that the goal was to demonstrate AI's potential as a research collaborator, not to deceive. They released the AI-generated code and datasets openly, inviting scrutiny and replication, which is a positive step toward accountability.

This incident also raises questions about the future of academic publishing. If AI tools become commonplace, how will journals adapt? Already, some outlets like Nature have guidelines requiring disclosure of AI assistance, while others ban it outright for certain elements like figure generation. There's a risk of flooding review systems with low-quality AI outputs, overwhelming volunteers and diluting standards. Conversely, AI could help by automating routine reviews or suggesting improvements, potentially making science more efficient.

In the broader context of Japan's AI ambitions, Sakana's work aligns with national efforts to catch up in the global AI race. With government initiatives pouring billions into AI R&D, startups like Sakana are seen as flagbearers, blending Eastern philosophies of harmony and evolution with Western tech innovation. Yet, this episode underscores a universal truth: in AI, as in science, claims must be tempered with evidence and context.

Ultimately, Sakana's experiment is a fascinating glimpse into a hybrid future where humans and machines co-create knowledge. It didn't quite achieve the fully autonomous breakthrough touted in headlines, but it advances the conversation on AI's role in research. As the field progresses, distinguishing hype from reality will be crucial to harnessing these tools responsibly. Whether this paves the way for AI scientists or merely enhances human ones, the journey is just beginning, promising both wonders and challenges ahead. (Word count: 1,128)

Read the Full TechCrunch Article at:
[ https://techcrunch.com/2025/03/12/sakana-claims-its-ai-paper-passed-peer-review-but-its-a-bit-more-nuanced-than-that/ ]

N※N

Sakana claims its AI-generated paper passed peer review a" but it's a bit more nuanced than that | TechCrunch

Sakana AI's Bold Claim: An AI-Generated Paper 'Passes' Peer Review, But the Reality is Far More Complex