Secure your AI models with OpenSSF Model Signing (OMS) 
Learn how OpenSSF’s AI/ML WG designed OMS to build trust in ML model artifacts.
https://openssf.org/blog/2025/06/25/an-introduction-to-the-openssf-model-signing-oms-specification/
#aisecurity
0 posts0 participants0 posts today
Hi y'all! New to infosec.exchange!
We're RSOLV - building automated security vulnerability detection + remediation (yes, a _fix_, not just a red flag)
While researching AI-generated code, we discovered something wild: 19.6% of AI package suggestions don't exist. Hackers are pre-registering them.
Traditional scanners miss this completely. We detect AND fix it.
Indie HackersBuilt the wrong thing at the wrong time - but discovered something worse (or "better")# Built the wrong thing at the wrong time - but discovered something worse (or better?) Hey IndieHackers! 👋 **TL;DR: Got my lunch eaten by Claude Code,...
Hello World! #introduction
Work in cybersec for 25+ years. Big OSS proponent.
Latest projects:
VectorSmuggle is acomprehensive proof-of-concept demonstrating vector-based data exfiltration techniques in AI/ML environments. This project illustrates potential risks in RAG systems and provides tools and concepts for defensive analysis.
https://github.com/jaschadub/VectorSmuggle
SchemaPin protocol for cryptographically signing and verifying AI agent tool schemas to prevent supply-chain attacks (aka MCP Rug Pulls).
https://github.com/ThirdKeyAI/SchemaPin
GitHubGitHub - jaschadub/VectorSmuggle: Testing platform for covert data exfiltration techniques where sensitive documents are embedded into vector representations and tunneled out under the guise of legitimate RAG operations — bypassing traditional security controls and evading detection through semantic obfuscation.Testing platform for covert data exfiltration techniques where sensitive documents are embedded into vector representations and tunneled out under the guise of legitimate RAG operations — bypassing...
New research tests on various models find that the #ChatGPTo3 model resists shutdown despite explicit instructions.
Read: https://hackread.com/chatgpt-o3-resists-shutdown-instructions-study/
Hackread - Latest Cybersecurity, Hacking News, Tech, AI & Crypto · ChatGPT o3 Resists Shutdown Despite Instructions, Study ClaimsFollow us on Bluesky, Twitter (X), Mastodon and Facebook at @Hackread
AI-powered features are the new attack surface! Check out our new blog in which LMG Security’s Senior Penetration Tester Emily Gosney @baybedoll shares real-world strategies for testing AI-driven web apps against the latest prompt injection threats.
From content smuggling to prompt splitting, attackers are using natural language to manipulate AI systems. Learn the top techniques—and why your web app pen test must include prompt injection testing to defend against today’s AI-driven threats.
LMG SecurityAre Your AI-Backed Web Apps Secure? Why Prompt Injection Testing Belongs in Every Web App Pen Test | LMG SecurityDiscover how prompt injection testing reveals hidden vulnerabilities in AI-enabled web apps. Learn real-world attack examples, risks, and why your pen test must include LLM-specific assessments.
Continued thread
Thanks to Kyle Wiggers for this article. We're honored to see our research covered by TechCrunch. 
Read the article here: https://techcrunch.com/2025/05/08/asking-chatbots-for-short-answers-can-increase-hallucinations-study-finds/
TechCrunch · Asking chatbots for short answers can increase hallucinations, study finds | TechCrunchTurns out, telling an AI chatbot to be concise could make it hallucinate more than it otherwise would have.
Microsoft Copilot for SharePoint just made recon a whole lot easier. 
One of our Red Teamers came across a massive SharePoint, too much to explore manually. So, with some careful prompting, they asked Copilot to do the heavy lifting...
It opened the door to credentials, internal docs, and more.
All without triggering access logs or alerts.
Copilot is being rolled out across Microsoft 365 environments, often without teams realising Default Agents are already active.
That’s a problem.
Jack, our Head of Red Team, breaks it down in our latest blog post, including what you can do to prevent it from happening in your environment.
Read it here: https://www.pentestpartners.com/security-blog/exploiting-copilot-ai-for-sharepoint/
Replied in thread
The assertion is, that one can not guarantee safety of #AI #LLM Models at all.
I'd like to get to the bottom of this, not that I doubt it by #AIsecurity is going to be increasingly more important, especially now that Old school #infosec has proven to be susceptible to the "Walk in and Seize control" exploit 
Edit;
Sometimes my friends tell me; Wulfy; /no_think
How Google DeepMind’s CaMeL Architecture Aims to Block LLM Prompt Injections
#AI #LLMs #AISecurity #PromptInjection #GoogleDeepMind #Cybersecurity #AIResearch #CaMeL #LLMSecurity #AISafety
Researchers claim breakthrough in fight against AI’s frustrating security hole - In the AI world, a vulnerability called "prompt injection" has haunted dev... - https://arstechnica.com/information-technology/2025/04/researchers-claim-breakthrough-in-fight-against-ais-frustrating-security-hole/ #largelanguagemodels #promptinjections #machinelearning #googledeepmind #simonwillison #rileygooside #aisecurity #chatgpt #chatgtp #biz #google #ai
Ars Technica · Researchers claim breakthrough in fight against AI’s frustrating security hole
Man, this whole AI hype train... Yeah, sure, the tools are definitely getting sharper and faster, no doubt about it. But an AI pulling off a *real* pentest? Seriously doubt that's happening anytime soon. Let's be real: automated scans are useful, but they just aren't the same beast as a genuine penetration test.
Honestly, I think security needs to be woven right into the fabric of a company from the get-go. It can't just be an afterthought you tack on when alarms are already blaring.
Now, don't get me wrong, AI definitely brings its own set of dangers – disinformation is a big one that springs to mind. But here's the thing: we absolutely *have* to get our heads around these tools and figure them out. If we don't keep pace, we risk becoming irrelevant pretty quick.
So, curious to hear what you all think – where do the greatest pitfalls lie with AI in the security field? What keeps you up at night?
Cloudflare turns AI against itself with endless maze of irrelevant facts - On Wednesday, web infrastructure provider Cloudflare announced a new featu... - https://arstechnica.com/ai/2025/03/cloudflare-turns-ai-against-itself-with-endless-maze-of-irrelevant-facts/ #largelanguagemodels #machinelearning #aisecurity #cloudflare #biz #ai
Ars Technica · Cloudflare turns AI against itself with endless maze of irrelevant facts
The full agenda is now live on our website, and we're kicking things off in Barcelona with an incredible first day! Join in on training sessions on AI Whiteboard Hacking, Full-Stack Pentesting, and iOS and Andriod App Security on day 1.
https://owasp.glueup.com/event/owasp-global-appsec-eu-2025-123983/home.html
Hackers are taking action where others have failed—warning ransomware victims, exposing AI security flaws, and questioning biohacking ethics. Should policymakers pay more attention to hacker research? Listen now and decide.
youtu.be- YouTubeEnjoy the videos and music you love, upload original content, and share it all with friends, family, and the world on YouTube.
Continued thread
◆ Hallucination and factual accuracy
◆ Bias and fairness
◆ Resistance to adversarial attacks
◆ Harmful content prevention
The LLM Benchmark incorporates diverse linguistic and cultural contexts to ensure comprehensiveness, and representative samples will be open-source.
Read about our methodology, and early findings: https://gisk.ar/3CRFdeB
We will be sharing more results in the coming months
gisk.arGiskard announces a new LLM Evaluation Benchmark during the Paris AI SummitGiskard partners with Google DeepMind to launch an independent multilingual LLM benchmark, evaluating hallucinations and AI security risks.
Open Source AI Models are a growing cybersecurity risk.
Organizations are increasingly using AI models from repositories like Hugging Face and TensorFlow Hub—but are they considering the hidden cybersecurity risks? Attackers are slipping malicious code into AI models, bypassing security checks, and exploiting vulnerabilities.
New research shows that bad actors are leveraging open-source AI models to introduce backdoors, execute arbitrary code, and even manipulate model outputs. If your team is developing AI solutions, now is the time to secure your AI supply chain by:
Vetting model sources rigorously Avoiding vulnerable data formats like Pickle Using safer alternatives like Safetensors Managing AI models like any other open-source dependency
As AI adoption skyrockets, you must proactively safeguard your models against supply chain threats. Check out the full article to learn more: https://www.darkreading.com/cyber-risk/open-source-ai-models-pose-risks-of-malicious-code-vulnerabilities
www.darkreading.comOpen Source AI Models: Big Risks for Malicious Code, VulnsCompanies pursing internal AI development using models from Hugging Face and other repositories need to focus on supply chain security and checking for vulnerabilities.
Malicious Code Found in AI Models Shared on Hugging Face #AI ##Cybersecurity #HuggingFace #AISecurity #PickleSerialization #MachineLearning #Python #OpenSourceAI
https://winbuzzer.com/2025/02/09/malicious-code-found-in-ai-models-shared-on-hugging-face-xcxwbn/
Ready to Secure AI Systems? Join Our 3-Day Hands-On Training at OWASP Global AppSec EU 2025!
Dive into AI/ML Whiteboard Hacking with expert Sebastien Deleersnyder from May 26-28, 2025 in Barcelona.
Designed for AI engineers, developers, architects, and security professionals, this intermediate-level training will equip you with practical skills to identify AI-specific threats.
I am reading up on abliterations:
https://huggingface.co/blog/mlabonne/abliteration
Still trying to wrap my head around the consequences of this. But...
...I kinda feel like abliterations have implications also for prompt injections?
As in, it feels like abliterations could mean that it is simply impossible to secure an LLM from prompt injection?
I'm sure I am misunderstanding stuff here. Anyone any input on this?
huggingface.coUncensor any LLM with abliterationA Blog post by Maxime Labonne on Hugging Face
DeepSeek Exposed!
Open HTTP ports linked to publicly exposed database, accessible without any authentication at all, allowed full control over database operations - a good thing? NOT! Sensitive data out in the open.
https://www.wiz.io/blog/wiz-research-uncovers-exposed-deepseek-database-leak #WIZ #cybersecurity #AI #DeepSeek #database #chathistory #opensource #AIsecurity #security