الوضع في بيئة الاختبار ضروري: التخفيف من مخاطر تنفيذ التعليمات البرمجية في أنظمة الذكاء الاصطناعي العميلة

The rise of agentic AI systems, which autonomously generate and execute code based on user requests, introduces significant cybersecurity risks. This blog post, authored by John Irwin and Kai Greshake from NVIDIA, highlights the critical need for sandboxing AI-generated code to prevent remote code execution (RCE) vulnerabilities. The post emphasizes that traditional sanitization methods are insufficient to protect against sophisticated prompt injection attacks and provides a case study illustrating how an NVIDIA AI Red Team discovered and addressed a critical vulnerability in an AI-driven analytics pipeline. The authors argue that sandboxing is not an optional security enhancement but a necessary architectural principle for securing AI-powered workflows.

The Risk of Untrusted AI-Generated Code

Agentic AI systems translate user requests into executable code, often in Python, creating a direct pathway for attackers to inject malicious code. The core problem stems from treating AI-generated code as trusted while the underlying LLM (Large Language Model) is processing potentially untrusted input. This creates a scenario where crafted prompts can bypass security measures and lead to RCE. Key attack layers include:

• Guardrail Evasion: Circumventing initial prompt filters designed to restrict certain topics or actions.
• Input Preprocessing Manipulation: Coercing specific output formats to control extraction variables (e.g., data and plotting parameters) used in code generation.
• Malicious Code Generation: Tricking the LLM into generating harmful code snippets.
• Code Payload Injection: Injecting the final malicious payload to escape any existing code restrictions.
• Arbitrary Command Execution: Ultimately executing a Base64-encoded terminal command on the target machine.

The Inadequacy of Sanitization

Sanitization techniques, which involve filtering or modifying code before execution, are often implemented as a primary defense. However, the post argues that sanitization alone is inherently limited. Attackers can exploit:

• Trusted Library Functions: Using seemingly safe functions within trusted libraries to achieve malicious outcomes.
• Static Filter Evasion: Bypassing static filters through encoding or context manipulation.
• Runtime Behavior Manipulation: Exploiting dynamic behaviors during code execution that sanitization cannot predict.

The authors emphasize that containment is the only scalable and reliable solution. Sandboxing the execution environment prevents AI-generated code from impacting system-wide resources, significantly limiting the potential blast radius.

Case Study: RCE Vulnerability in an Analytics Workflow

The NVIDIA AI Red Team identified a Remote Code Execution (RCE) vulnerability, CVE-2024-12366, in an internal analytics workflow that utilized a third-party library to translate natural language queries into Python code. Despite the library’s initial sanitization measures, the team demonstrated how prompt injection payloads could bypass these controls. This exploit included namespace exposures, encoding bypasses, and context manipulation techniques that subverted code filters at runtime. The identification of this vulnerability underscored a critical flaw: Systems that execute AI-generated code without user-specific sandboxing remain vulnerable to control plane compromise. Sanitization, while a useful defense-in-depth strategy, is not sufficient for guaranteeing execution safety.

Sandboxing as the Essential Control

The blog post advocates for sandboxing as the primary control mechanism for mitigating risks associated with AI-generated code execution. Sandboxing provides a reliable boundary by isolating each execution instance, ensuring that any malicious or unintended code path is contained within a restricted environment. Following the reported disclosure, the maintainers of the third-party library implemented sandboxing.

Lessons for AI Application Developers

The post concludes with key lessons for AI application developers:

• Treat AI-generated code as inherently untrusted: Apply the same level of caution as with user-supplied inputs.
• Sanitization is defense-in-depth, not a primary control: Relying solely on sanitization creates a false sense of security. Consider using NVIDIA NeMo Guardrails.
• Execution isolation is mandatory: Use sandboxing or remote execution environments (e.g., AWS EC2 or Brev) to limit the impact of malicious code.
• Collaboration is critical: Participate in open disclosure processes to share security findings and contribute to collective mitigation efforts.

By embracing containment-first architectures, the industry can ensure that AI-driven innovation scales safely and securely.

Source: NVIDIA