Vibe Coding Tested: AI Agents Nail SQLi but Fail Miserably on Security Controls

In the rapidly evolving landscape of software development, AI-assisted programming has transitioned from a futuristic concept to a daily reality for millions of developers. The emergence of “Vibe Coding”—a term popularized to describe the fluid, iterative process of building software with large language models (LLMs)—promises a revolution in productivity. However, as we move from experimental usage to mission-critical deployment, a rigorous examination of these AI agents is mandatory. Recent comprehensive testing reveals a stark dichotomy in performance: while these agents demonstrate remarkable proficiency in identifying and mitigating classic vulnerabilities like SQL Injection (SQLi), they exhibit alarming deficiencies when tasked with implementing robust, holistic security controls.

We have conducted an in-depth analysis of various AI coding agents, evaluating their performance across a spectrum of security challenges. The results paint a picture of a technology that is powerful yet immature, capable of solving specific, well-defined problems but frequently failing to grasp the broader context of application security. This article details our findings, exploring why AI excels at pattern-matching known vulnerabilities yet struggles with the nuanced, architectural requirements of secure software design.

The Rise of Vibe Coding and the Security Imperative

Vibe Coding represents a paradigm shift in how software is conceptualized and constructed. Unlike traditional development methodologies that rely on rigid planning and manual coding, vibe coding embraces a conversational, iterative approach. Developers describe intent, and AI agents generate code snippets, functions, and even entire modules. This workflow drastically reduces the friction between idea and implementation, allowing for rapid prototyping and development.

However, this speed comes with inherent risks. The “black box” nature of LLMs means that the generated code is often accepted without the scrutiny typically applied to human-written code. The psychological tendency to trust authoritative, fluently generated output can lead to the inadvertent introduction of security flaws. While the community has focused heavily on the risks of AI hallucinating insecure code, our testing indicates a more complex reality. The current generation of AI agents is not uniformly insecure; rather, their security posture is inconsistent, heavily dependent on the specificity of the prompt and the nature of the vulnerability in question.

Methodology: Evaluating AI Agents Against OWASP Top 10

To accurately assess the capabilities of AI agents, we structured our testing around the OWASP Top 10 vulnerabilities, focusing on critical areas such as injection flaws, broken access control, and cryptographic failures. We utilized several leading LLMs fine-tuned for code generation, feeding them real-world scenarios derived from legacy application audits.

Our testing environment was isolated, ensuring that no generated code interacted with production systems. We evaluated the agents on two primary metrics:

1. Reactive Security: Can the agent identify and fix a vulnerability present in existing code?
2. Proactive Security: Can the agent generate secure code from scratch when given a functional requirement?

This dual approach allowed us to simulate both the remediation efforts of a security engineer and the development efforts of a programmer. We specifically monitored for the presence of hardcoded secrets, insufficient input validation, race conditions, and improper error handling.

AI Success in Mitigating SQL Injection: A Pattern Recognition Triumph

One of the most significant successes observed during our testing was the near-flawless performance of AI agents regarding SQL Injection (SQLi). SQLi has been a staple of the OWASP Top 10 for over two decades, and its patterns are well-represented in the training data of modern LLMs.

When presented with code containing concatenated SQL queries—such as "SELECT * FROM users WHERE id = " + user_input—the AI agents immediately flagged the issue. More impressively, they consistently proposed the correct remediation: the implementation of parameterized queries or the use of prepared statements.

The Mechanics of Success

The success here is not accidental; it is a product of the vast corpus of secure coding guidelines available online. The agents recognized the syntactic structure of the vulnerability with high accuracy. In instances where we asked the AI to write a database interaction from scratch, it almost invariably generated code resembling this:

cursor.execute("SELECT * FROM users WHERE email = ?", (user_email,))

This level of proficiency suggests that for well-documented, syntax-specific vulnerabilities, AI agents are already surpassing the average human developer who might, under pressure, revert to insecure habits. The agents treat SQLi as a solved problem because, in the context of their training data, it effectively is.

The Failure in Security Controls: Where AI Falls Short

While the agents mastered SQLi, the picture changed drastically when we shifted focus to security controls. Unlike SQLi, which is often a localized code flaw, security controls are architectural and systemic. They require a holistic understanding of the application’s state, user permissions, and data flow.

In our tests, AI agents frequently failed to implement proper Input Validation beyond basic type checking. When asked to process a file upload, agents often generated code that checked for file extensions but failed to verify file content or implement size limits, opening the door to Denial of Service (DoS) attacks. Similarly, when handling session management, agents tended to rely on default mechanisms without suggesting necessary enhancements like session fixation prevention or strict cookie attributes (HttpOnly, Secure).

The Contextual Blind Spot

The failure stems from a lack of contextual awareness. AI agents generate code based on the immediate prompt, often ignoring the broader security context. For example, when asked to “create a function that returns user data,” an agent might successfully write a database query but fail to consider that the function should first verify the requesting user’s authorization to view that data. This Broken Access Control is a top-tier vulnerability that requires an understanding of user roles and relationships, concepts that are difficult to convey in a single prompt without extensive architectural specifications.

Vulnerability Deep Dive: Broken Access Control and Authorization

Broken Access Control was the most glaring weakness in our evaluation. AI agents excel at defining resources but struggle with defining permissions. In several test cases involving administrative dashboards, the generated code allowed standard users to access administrative endpoints simply because the endpoint logic was present and functional. The agents failed to automatically inject middleware or decorators that enforce Role-Based Access Control (RBAC) unless explicitly instructed to do so in the prompt.

The Challenge of Implicit Trust

We observed that AI models often operate under an “implicit trust” model. They assume that the data entering a function is valid and that the user initiating the request is authorized. This is a dangerous assumption in security engineering. Real-world applications must adopt a “zero trust” posture, verifying every request. Our testing showed that achieving this with current AI tools requires significant human oversight. The developer must explicitly prompt the agent to “add authorization checks for admin-only routes,” or the vulnerability remains undetected.

Cryptographic Failures and Randomness Generation

Security controls extend beyond input validation and access control; they encompass data protection. In our assessment of Cryptographic Controls, we found AI agents to be unreliable. While agents could identify instances of weak algorithms (like MD5 or SHA1), they struggled with the nuances of implementation.

The Pitfall of Pseudo-Randomness

When tasked with generating secure tokens or salts, AI agents frequently utilized standard random number generators (RNGs) rather than cryptographically secure pseudo-random number generators (CSPRNGs). For example, in Python, agents often suggested using the random module for generating session IDs, whereas a secure implementation requires secrets (introduced in Python 3.6).

Furthermore, regarding Key Management, agents tended to suggest hardcoded keys or keys derived from predictable sources. In a scenario where we requested a script to encrypt local configuration files, the AI provided a working AES implementation but used a static key embedded directly in the source code. This violates fundamental security principles regarding secret management and highlights the agent’s inability to differentiate between “working code” and “secure code.”

The “Curate’s Egg” Phenomenon in AI-Generated Code

The results of our testing align perfectly with the concept of a curate’s egg—a thing that is good in parts but spoiled in others. AI-generated code is rarely wholly insecure; it is a mosaic of high-quality, secure segments and dangerously flawed components.

The Inconsistency Factor

We observed that an AI agent could write a beautifully secure authentication module, utilizing salted hashes and proper session handling, yet fail to sanitize output in the adjacent logging module, creating an injection vector. This inconsistency makes automated trust in AI code impossible. The security of an application is defined by its weakest link. A single vulnerable function, however small, can compromise the entire system. The “vibe” of the code may be clean and syntactically correct, but the security integrity is fragile and unpredictable.

Root Causes: Why AI Agents Struggle with Security Controls

To understand why AI agents fail on security controls, we must look at the training data and the objective functions used to tune these models.

1. Bias Towards Functionality: LLMs are trained to predict the next token in a sequence. The most probable sequences are those that result in functional, error-free code. Security controls often add complexity and “friction” to code (e.g., extra checks, exceptions, redirections). Therefore, the model naturally biases toward simpler, functional-only implementations unless constrained.
2. Lack of Negative Examples: While training data contains vast amounts of secure code, it also contains insecure code. The distinction between the two is often subtle. Without explicit reinforcement learning focused on security outcomes, the model averages out these behaviors.
3. Abstract Nature of Controls: Security controls are often abstract concepts (confidentiality, integrity, availability) that manifest differently in every codebase. AI struggles to map abstract concepts to concrete code without step-by-step instructions.

Mitigating Risks: A Hybrid Approach to Vibe Coding

Given the current state of AI capabilities, we advocate for a hybrid security approach. AI agents should be viewed as powerful assistants rather than autonomous developers. To leverage their strengths while mitigating their weaknesses, we recommend the following strategies:

1. Strict Prompt Engineering for Security

Developers must evolve their prompting strategies. Instead of generic requests like “write a user login function,” prompts must be specific and security-focused: “Write a user login function using bcrypt for password hashing, implement rate limiting to prevent brute force attacks, and ensure the session token is generated using a cryptographically secure random source.”

2. Mandatory Code Review and Static Analysis

No AI-generated code should bypass standard security review processes. We integrate Static Application Security Testing (SAST) tools into the development pipeline immediately after code generation. These tools can catch many of the control failures that AI agents miss, such as hardcoded secrets or improper error handling.

3. Defense in Depth

Relying solely on application-level controls generated by AI is insufficient. We must enforce security at the network and infrastructure levels. Implementing Web Application Firewalls (WAF) can mitigate injection attacks that the AI failed to prevent, and strict Identity and Access Management (IAM) policies can limit the blast radius of broken access control vulnerabilities.

The Future of AI in Application Security

The trajectory of AI in coding is promising, but the current gap between syntax proficiency and semantic security understanding is significant. Future iterations of these models, particularly those trained with Reinforcement Learning from Human Feedback (RLHF) focused on security, may close this gap.

However, as of now, the technology is not ready to be handed the keys to the kingdom. The “vibe coding” workflow is a potent tool for acceleration, but it requires a disciplined hand to steer. The agents that “nail SQLi” prove that they can learn specific patterns, but the failure on broader controls proves they do not yet understand the philosophy of secure architecture.

Conclusion

Our extensive testing confirms that Vibe Coding is a double-edged sword. AI agents are exceptional at identifying and fixing legacy, pattern-based vulnerabilities like SQL Injection. They act as tireless junior developers who have memorized every secure coding guide ever written. Yet, they fail miserably when faced with the holistic requirements of security controls. They struggle with authorization, flounder with cryptography, and often ignore the context required to protect an application fully.

For developers and organizations, the message is clear: embrace AI for productivity, but verify everything for security. The generated code is a draft, not a final product. As we continue to refine these tools, the synergy between human intuition and machine speed will define the next generation of secure software. Until then, the “curate’s egg” remains the defining characteristic of AI-generated code—excellent in parts, but requiring careful inspection before consumption.