Shockproofing AI: How higher ed can deploy secure AI models and agents

AI is transforming higher education — and educators and administrators alike are finding it difficult to keep up. Streamlined business operations, easy lesson planning, and student learning assistance are just some of the unprecedented opportunities AI promises. But there are security risks associated with this technology, and for institutions entrusted with sensitive data and upholding academic integrity, the stakes couldn’t be higher.

What makes AI so risky? It introduces new factors like probability drift, autonomy, and dynamic behavior that do not fit neatly within the protection found in the controls of traditional security frameworks. According to MIT Sloan (2024), 71% of organizations deploying AI without strong oversight experienced at least one significant incident within a year. With funding uncertainties and competition at an all time high, no college or university can afford to be part of that statistic.

The opportunity—and the risk

AI offers three major transformative benefits for higher education.

• Workforce performance: Helping faculty and staff deliver higher-quality output in less time.
• Operational efficiency: Automating and tackling repetitive tasks so teams can focus on strategic priorities.
• Enhanced experiences: Powering personalized learning and responsive student services.

But these benefits hinge on one critical resource: data. Data fuels AI models, yet data security remains the least mature area in cybersecurity. When sensitive academic, financial, or research data enters an AI model without proper safeguards, the consequences can be catastrophic.

Let’s take a closer look at the specific risks associated with AI use.

Risk #1: Inadequate access controls

GenAI applications often lack proper permission checks, resulting in confidential documents being retrieved without verifying user access. There can be a human element to this as well. Even if the GenAI application does properly conduct permission checks, many institutions will provide broad access regardless of the individual’s role. This can give advanced permissions to AI models or agents that do not have the same oversight that human users have.  That could lead to circumventing controls to access and share confidential documents in their AI queries.

Risk #2: Retrieval-Augmented Generation (RAG) vulnerabilities

RAG is a technique for supporting large language models (LLMs) by allowing them to answer questions with dynamic data (e.g. a database that is regularly updated) and makes using the AI model more efficient from a cost and operations perspective without having to train the AI model to all of the changes.

One potential attack would be an indirect prompt injection where the source of RAG had malicious code that was digested.  For example, a hyperlink that the AI model consumes that includes malware.  This can lead to the manipulation of the system’s decision-making, the distribution of disinformation to the user, the disclosure of sensitive information, the orchestration of intricate phishing attacks, and the execution of malicious code.

Risk #3: Open source libraries

Open source LLMs are fully accessible for anyone to use, modify, and distribute. This transparency allows for extensive customization — and extensive risks. Open source libraries can be vulnerable to misinformation, phishing attempts, bias or discrimination, intellectual property theft, and data leaks. When testing these libraries, 23% showed evidence of tampering, potentially introducing backdoors or vulnerabilities.

Risk #4: Shadow AI

Unauthorized LLM usage — what is sometimes called shadow AI — is also a common risk factor. In fact, 40–65% of employees have admitted to inputting company data into an LLM without manager knowledge. In a higher ed setting, this could translate to sensitive financial or student data being inputted into an LLM, which could have ethical and regulatory consequences.

Defining trustworthy AI

Once you fully understand the AI cybersecurity risks, the next step is to define what trustworthy AI means for your college or university. Here are some guiding principles for trustworthy AI to get you started.

• Safe and secure: The technology is resistant to misuse and attacks.
• Ethical and fair: The technology avoids bias and aligns with social values.
• Accountable: There is clear responsibility for AI decisions. Human oversight is helpful to keep AI decisions accountable.
• Transparent and explainable: The AI outputs are auditable. If you can’t understand the how and why behind AI results, then it isn’t a good idea to rely on them. 
• Resilient and adaptable: The technology handles unexpected scenarios without cascading failures.

As we look at global requirements on safe AI use, all of the above our commonly referenced and hope will be considered in the recent White House announcement for a federal set of requirements here in the United States.

Creating a blueprint for AI security

A modern AI security framework should include several guardrails to mitigate risk. Let’s first explore how you can combat the threats Retrieval-Augmented Generation (RAG) introduces.

You can mitigate several AI vulnerabilities by:

• Sandboxing your LLM. An LLM sandbox environment is an isolated, controlled space where AI-generated content can be executed safely without compromising the broader system or exposing sensitive data.
• Separating and identifying external content. Clearly denote untrustworthy content — this will limit its influence on user prompts.
• Requiring human-in-the-loop.. For high-risk behavior or questions that reference private data, it is a good idea to have human oversight built in as a step-up escalation policy.

In addition to AI-specific protection, many cybersecurity best practices should also be applied to the use of models and agents.

• Develop real-time trust signals instead of once-a-year reports. This will ensure that any issues are caught and solved quickly.
• Create simplified and accurate reporting between external stakeholders so that everyone understands how the system is operating.
• Conduct pre-execution filtering — detect malicious intent via pattern matching and semantic analysis. (Semantic analysis is the process of understanding the intent and context behind queries to more easily identify if the user is in search of proprietary information or trying to reveal security vulnerabilities.)
• Moderate outputs, scan for personally identifiable information and enforce policies.

If these processes seem complex, that’s because they are! While it is important for institutions to understand the basics of AI security frameworks, it can be beneficial to work with partners to manage the day to day operations. This way there is an expert always on the case, keeping things secure while your faculty and staff focus on their main responsibility: providing impactful educational experiences for your students.

Next steps for higher education leaders

As AI reshapes higher education, it’s critical that institutions prioritize trustworthy AI principles. By implementing robust security frameworks, colleges and universities can harness innovation without compromising integrity or data protection. The future of AI security will be built on trust — earned daily, even when no one is watching. Take steps now to create a secure, ethical, and resilient academic environment.

Interested in learning more? Cloud Security Alliance provides a free, open controls framework (AI Controls Matrix) and a library of free research that you can download to measure your security practices along your journey into AI models and agents on your campus.  We also provide security training and certification for those students or university professionals that want to better understand how to implement and maintain robust AI security on your campus.