The Ghost Forts in Your API Perimeter
Imagine a medieval castle. The walls are high, the gates guarded, and the patrols regular. But over the years, old watchtowers and abandoned outposts—ghost forts—were left to crumble. Attackers don't need to breach the main gate; they just slip into one of those forgotten towers. Your API landscape is no different. As teams push code rapidly, APIs are created, deprecated, and sometimes simply abandoned—still listening, still accessible, and entirely unmanaged. This is API sprawl, and it creates ghost forts inside your perimeter.
Ghost forts are endpoints that were never formally documented, APIs that were spun up for experiments and never taken down, or third-party integrations that were left active after the project ended. They exist outside your regular governance processes, often without authentication, rate limiting, or monitoring. Many industry surveys suggest that organizations have 2x to 3x more APIs than they think they do. For example, a typical e-commerce company might have a handful of official REST endpoints but dozens of internal microservice APIs, legacy SOAP services, and experimental GraphQL endpoints that were never cleaned up. Each of these is a potential entry point for attackers.
The core problem is that modern development encourages speed over governance. CI/CD pipelines make it easy to deploy new endpoints in minutes, but there's no automatic process to remove them when they're no longer needed. Meanwhile, the perimeter has shifted. APIs are exposed to the internet, partners, and mobile apps. You cannot secure what you don't know exists. The stakes are high: a single unmonitored API can lead to data breaches, compliance violations, or financial loss. To fix this, you need to think like a military strategist surveying a forgotten battlefield. You must map every structure, identify the weak points, and systematically demolish the ghost forts.
In this guide, I'll walk you through a practical approach to auditing API sprawl. We'll start with understanding the nature of ghost forts, then move to discovery, assessment, and finally remediation. By the end, you'll have a repeatable process to keep your API perimeter clean and secure. Let's begin by exploring why these ghost forts form in the first place.
Why Ghost Forts Form: The Sprawl Cycle
API sprawl doesn't happen overnight. It's a gradual accumulation driven by several common patterns. First, there's the prototype that went to production. A developer builds a quick API for a proof-of-concept, it works, and it never gets refactored or cleaned up. Second, there's the forgotten integration. You connect to a third-party service for a limited-time campaign, but the API key and endpoint remain active years later. Third, there's the legacy service that's still running but no one uses. A microservice from three years ago might still be listening on an internal port, but its documentation is lost. These patterns are amplified by team turnover and lack of centralized API management.
The Attacker's Perspective
Attackers love ghost forts because they are often less protected. They scan for common patterns like /api/v1/old, /experimental, or /internal. They look for endpoints that return verbose error messages or don't require authentication. A single forgotten endpoint with a SQL injection vulnerability can compromise your entire database. Understanding this mindset is key to prioritizing your audit. You must think like an attacker to find the weak points before they do.
Mapping the Battlefield: How to Discover Your API Surface
Before you can eliminate ghost forts, you need to know where they are. API discovery is the process of creating a comprehensive inventory of every endpoint that touches your network—both internal and external. Think of it as mapping the terrain before a battle. You wouldn't charge into unknown territory without a map, and you shouldn't secure an API perimeter without knowing every entry point. In this section, I'll cover three main methods for API discovery: network traffic analysis, code repository scanning, and integration with API gateways. Each method has strengths and blind spots, so a combination is best.
Network traffic analysis is the most direct method. By capturing and analyzing traffic at your load balancers, firewalls, or using packet inspection tools, you can see every API call being made to and from your infrastructure. This includes endpoints that are not documented anywhere. Tools like Wireshark, tcpdump, or commercial API security platforms can help. For example, one team I read about set up a network tap on their internal switch and discovered 40 endpoints that no one knew existed. These were internal microservices that had been deployed years ago and were still responding to requests. The catch is that traffic analysis only shows endpoints that are actively being used. Dormant endpoints—ghost forts that no one calls—won't appear in traffic logs.
Code repository scanning is another powerful technique. By searching your version control systems (like Git) for API route definitions, you can find endpoints that are defined in code but may not be actively monitored. You can look for patterns like @RequestMapping, app.get(), or router.post() across all repositories. This helps you find endpoints that were created in code but never deployed, or that were deployed but never documented. However, this method can produce false positives, as some code may be experimental or never shipped. You'll need to cross-reference with actual traffic.
API gateways, if you have one, are a goldmine. Every request that passes through the gateway is logged, and most gateways provide a dashboard of all registered endpoints. But the trick is that not all APIs go through the gateway. Legacy services or internal-only APIs might bypass it entirely. Therefore, relying solely on your gateway will miss a significant portion of your surface. A multi-pronged approach is essential: use the gateway for the known, traffic analysis for the active, and code scanning for the potential. Once you have a list, you need to categorize and prioritize.
Practical Steps for Mapping
Start by exporting a list of all endpoints from your API gateway. Then, set up a network traffic capture for 24-48 hours to log all API calls. Use a tool like mitmproxy or a cloud-based API security solution. Next, run a script across all your Git repositories to extract route definitions. Merge these three lists, remove duplicates, and you'll have a baseline. Expect surprises—many teams find 30-50% more endpoints than they thought existed. The next step is to assess each endpoint's risk.
Assessing the Forts: Risk Prioritization for Forgotten APIs
Once you have a list of every API endpoint, you need to assess which ones pose the greatest risk. Not all ghost forts are equally dangerous. Some may be internal-only and heavily firewalled, while others may be exposed to the public internet with weak authentication. Risk assessment is about understanding three dimensions: exposure, sensitivity, and vulnerability. In this section, I'll walk you through a repeatable framework for scoring each endpoint and deciding which to decommission first.
Exposure refers to how accessible the endpoint is. An endpoint that is publicly routable on the internet is a higher priority than one that's only accessible from within a private subnet. You can determine exposure by reviewing network configurations, firewall rules, and DNS records. For example, if an API has a public DNS record like api-old.example.com, it's exposed. Even if it's behind a VPN, it's still more exposed than a service that only listens on localhost. Sensitivity is about what data the API handles. An endpoint that returns customer PII, payment details, or internal configuration secrets is high sensitivity. Low-sensitivity endpoints might return public data or server health metrics. You can gauge sensitivity by reading the API's response schema or by observing traffic patterns.
Vulnerability is the hardest to assess without a full penetration test, but you can make educated guesses. APIs that are old, written in deprecated frameworks, or lack authentication are likely more vulnerable. Also, APIs that return verbose error messages (like stack traces) are easier to exploit. Combine these three factors into a simple scoring system: give each endpoint a score from 1 to 5 for exposure, sensitivity, and vulnerability, then multiply or add them to get a priority score. For instance, an exposed, high-sensitivity, high-vulnerability endpoint gets a score of 15 (5+5+5) and should be addressed immediately. An internal, low-sensitivity, low-vulnerability endpoint might score 3 and can wait.
But scoring is just the start. You also need to consider business context. An endpoint that is critical to a revenue-generating process cannot simply be shut down without planning. In such cases, you might need to refactor or migrate it to a managed gateway instead of decommissioning. The key is to create a prioritized list that balances risk with business impact. Use a simple matrix: high risk + low business impact = decommission first; high risk + high business impact = migrate or secure urgently; low risk + low business impact = decommission when convenient; low risk + high business impact = monitor but don't block.
Common Prioritization Mistakes
A common mistake is to focus only on externally exposed endpoints. Internal APIs can be just as dangerous if an attacker gains a foothold elsewhere. Another mistake is to ignore endpoints that appear to be inactive. Dormant endpoints can be reactivated by attackers who discover them. Always assume that any endpoint that exists is a potential risk. Finally, don't rely solely on automated scores. Human judgment is needed to interpret edge cases, such as an internal API that handles highly sensitive data.
Tools of the Trade: Comparing API Discovery and Security Solutions
There are many tools available to help with API discovery and security, but they vary widely in capabilities and cost. Choosing the right one depends on your organization's size, budget, and technical depth. In this section, I'll compare three common approaches: manual auditing with open-source tools, commercial API discovery platforms, and API gateways with built-in governance. I'll cover pros, cons, and best-use scenarios so you can decide what fits your battlefield.
Manual auditing using open-source tools is the most flexible and cost-effective approach, but it requires significant effort. You can use Wireshark or tcpdump for traffic capture, grep and regex for code scanning, and a spreadsheet to track findings. This approach is ideal for small teams with fewer than 50 endpoints and strong scripting skills. The main advantage is control—you can customize every step. The disadvantage is that it's time-consuming and error-prone. You might miss endpoints that are rarely called or hidden in obscure repositories. Also, maintaining the inventory manually is a burden. For example, a startup I read about used this method and found 20 ghost endpoints, but they had to repeat the process every quarter, which took two engineers a full week each time.
Commercial API discovery platforms (like Noname Security, Salt Security, or Traceable) automate much of the process. They integrate with network traffic, cloud APIs, and code repositories to build a living inventory. They also provide risk scoring, behavioral analysis, and sometimes automatic blocking. These tools are excellent for organizations with hundreds or thousands of APIs. The pros are speed, accuracy, and continuous monitoring. The cons are cost (often $10,000+ per year) and complexity of deployment. Some tools require agents or network taps, which may not be feasible in all environments. For example, a mid-sized fintech company deployed a commercial platform and reduced their discovery time from weeks to hours, but they struggled with false positives from legacy traffic.
API gateways with governance features (like Kong, Apigee, or AWS API Gateway) are a middle ground. They can enforce authentication, rate limiting, and logging for all APIs that pass through them. However, they only cover APIs that are consciously routed through the gateway. They won't discover shadow APIs that bypass the gateway. The advantage is that they are often already deployed for other reasons, so there's no additional cost. The disadvantage is that they don't solve the discovery problem for ghost forts. You'll still need a separate discovery process. In practice, many organizations use a combination: a gateway for known APIs and a discovery tool for unknowns.
Choosing the Right Approach
Start with a manual audit to understand the scale of your problem. If you find more than 50 endpoints, consider a commercial tool. If you have a gateway, use it as a foundation but don't rely on it exclusively. Budget for ongoing discovery as a recurring task, not a one-time project. Remember, the goal is not just to find ghost forts once, but to prevent new ones from forming.
Growth Mechanics: Preventing Sprawl Through API Governance
Auditing and remediating ghost forts is a one-time exercise, but preventing new ones from forming requires ongoing governance. In this section, I'll discuss how to build processes and culture that prevent API sprawl from recurring. Think of it as establishing a standing army to patrol your perimeter. Without governance, your ghost forts will return. The key elements are: an API registry, CI/CD checks, lifecycle management, and team education.
An API registry is a centralized database of all approved APIs, their owners, and their lifecycle status. When a new API is created, it must be registered before it can be deployed. This can be enforced through a policy in your CI/CD pipeline. For example, a deployment script can check if the API route is already in the registry. If not, the deployment fails. This forces developers to document their APIs from the start. The registry should include metadata: endpoint path, method, owner, data sensitivity, authentication requirements, and expiration date. For internal APIs, you can set a default expiration of 12 months, after which the API must be renewed or it's automatically decommissioned. This prevents indefinite ghost forts.
CI/CD checks are the enforcement mechanism. In your pipeline, add a step that scans the code for new API routes and verifies they exist in the registry. If not, the build fails with an error message instructing the developer to register the API. This can be implemented as a simple script using your registry's API. Additionally, you can add a step that checks for deprecated APIs in the codebase and flags them for removal. Over time, this creates a culture where API governance is part of the development workflow, not an afterthought.
Lifecycle management is about explicitly defining stages for every API: design, development, active, deprecated, and decommissioned. Each stage has clear criteria and actions. For example, when an API is deprecated, you should set a sunset date, notify all consumers, and schedule its removal. The registry should automatically send reminders to the owner when the sunset date approaches. After the date, the API can be automatically taken down. This prevents the common pattern where an API is deprecated but never removed. For external APIs, you may need to maintain backward compatibility for longer, but the same principle applies.
Team education is the human element. Developers need to understand why ghost forts are dangerous. Hold a brown-bag session where you walk through a real incident (anonymized) of a forgotten API that led to a breach. Show them the discovery process and the risk scores. Encourage them to adopt the mindset of an attacker. When developers feel ownership over their APIs, they are more likely to clean up after themselves. Also, consider adding a "cleanup day" every quarter where teams review and retire old APIs. This makes governance a shared responsibility.
Measuring Success
Track key metrics: number of ghost forts found in each audit, time to decommission, and percentage of APIs registered. Over time, these numbers should improve. If they don't, revisit your governance process. Remember, the goal is to make the process frictionless so that developers follow it willingly, not because they are forced.
Pitfalls and Mistakes: What Can Go Wrong During an API Audit
Even with the best intentions, API audits can fail. Common pitfalls include incomplete coverage, false positives, analysis paralysis, and lack of executive support. In this section, I'll outline these mistakes and how to avoid them, based on common experiences shared by practitioners. Learning from these mistakes can save you time and frustration.
Incomplete coverage is the most common mistake. Many teams rely solely on their API gateway for discovery, but as we've discussed, ghost forts often bypass the gateway. Similarly, scanning only production environments ignores staging and development endpoints that are also accessible. To avoid this, use multiple discovery methods: traffic analysis, code scanning, and gateway logs. Also, include all environments—production, staging, development—and all protocols (HTTP, gRPC, WebSocket). A thorough audit covers the entire attack surface.
False positives can overwhelm your team. For example, a code scan might find a route definition in a test file that never gets deployed. Or a traffic capture might include calls to a third-party service that you don't control. To manage this, require human verification for each endpoint. Cross-reference with deployment records, network logs, or direct probing. Create a scoring system that flags endpoints as "confirmed" or "suspected." Focus your energy on confirmed endpoints first. For suspected ones, do a quick manual check (e.g., send a test request) before adding them to the inventory.
Analysis paralysis happens when you have a huge list of endpoints and don't know where to start. This is where prioritization is critical. Use the risk scoring system I described earlier. Don't try to fix everything at once. Instead, do a phased approach: first, decommission the high-risk, low-impact endpoints. Then, secure the high-risk, high-impact ones. Finally, clean up the rest. Set a timeline of 4-6 weeks for the initial audit and remediation. After that, move to ongoing monitoring.
Lack of executive support can kill your initiative. API sprawl is often invisible to leadership, who may see it as a technical debt issue with no immediate business impact. To get buy-in, quantify the risk. For example, you can present a scenario: "We have 50 unmanaged APIs; if any one of them is compromised, the average cost of a data breach is $4 million." Use industry benchmarks (without fabricating numbers) to make your case. Also, tie it to compliance requirements like PCI-DSS or SOC 2, which mandate asset inventories. Once you have support, communicate progress regularly.
Other Common Mistakes
Don't forget to secure the tools you use for the audit themselves. A scanning tool with weak authentication can become a new ghost fort. Also, avoid alert fatigue by setting up proper thresholds. Finally, don't assume that once you've cleaned up, the problem is solved. Sprawl is a continuous challenge; you need ongoing audits.
Frequently Asked Questions About API Ghost Forts
This section answers common questions that arise when teams start auditing their API perimeter. These are based on questions I've seen in forums and practitioner discussions. Each answer provides practical guidance to help you move forward.
1. How often should I audit for ghost forts? At least once a year, but more frequently if you have a fast-moving development culture (quarterly). After the initial cleanup, you can reduce frequency, but always run a full audit after major infrastructure changes or acquisitions.
2. What if I find an API that no one owns? Mark it as "unowned" and try to trace its origin through code commits or deployment logs. If you can't find an owner, treat it as high risk and decommission it after a 30-day grace period with a warning to internal consumers. If no one complains, it's safe to remove.
3. Can I automate the decommissioning process? Yes, but be cautious. Automated decommissioning can break services that still depend on the API. Start with manual decommissioning for the first few rounds. Once you have a reliable registry and dependency mapping, you can automate the removal of endpoints that have been deprecated for a set period and have no traffic for 90 days.
4. How do I handle APIs that are exposed to partners? Partner APIs are especially sensitive because they are outside your direct control. You should have a separate inventory for partner-facing APIs and enforce strict authentication (mTLS, OAuth). If a partner integration is no longer used, revoke the API keys and decommission the endpoint. Communicate changes through a formal sunset process.
5. What about APIs in serverless functions? Serverless platforms like AWS Lambda or Azure Functions can create many temporary endpoints. These are harder to track because they are ephemeral. Use platform-specific logging (CloudWatch, Azure Monitor) to capture invocation patterns. Also, enforce that every serverless function must be registered in your API registry before it can be deployed. Some serverless frameworks allow you to attach API Gateway, which can help with discovery.
6. Is it worth investing in a commercial tool for small teams? For small teams with fewer than 20 endpoints, manual methods are usually sufficient. However, if you expect growth or have compliance requirements, a commercial tool can save time. Many vendors offer free tiers or trials for small organizations. Evaluate them against your specific needs.
7. How do I convince my team to take this seriously? Lead by example. Run a small audit on a single service and present the results. Show them a ghost fort you found and demonstrate how easy it is for an attacker to exploit it. Use real-world analogies (like the ghost forts in this guide). Make it a game or competition—who can find the oldest forgotten API? A little friendly competition can build momentum.
Securing Your Perimeter for the Long Term
Auditing your API perimeter is not a one-time project; it's an ongoing discipline. In this final section, I'll summarize the key takeaways and provide a concrete action plan you can start tomorrow. The goal is to move from a state of ignorance to one of continuous awareness, where ghost forts are discovered and eliminated before they can be exploited.
First, remember the three-step cycle: discover, assess, remediate. Discover your entire API surface using multiple methods. Assess each endpoint's risk using exposure, sensitivity, and vulnerability. Remediate by decommissioning, securing, or migrating endpoints based on priority. After the initial cleanup, establish governance to prevent new ghost forts: an API registry, CI/CD checks, lifecycle management, and team education.
Second, choose tools that match your scale. Start with manual methods if you're small; invest in automation as you grow. Use your API gateway as a foundation but don't rely on it exclusively. Remember that the most important tool is a culture of awareness. When every developer thinks about the security of their APIs, sprawl decreases naturally.
Third, don't let perfect be the enemy of good. You may never find every ghost fort, but finding and removing the high-risk ones dramatically reduces your attack surface. Celebrate small wins. After your first audit, you'll likely have a list of endpoints to decommission. Removing even one forgotten API with sensitive data is a victory.
Finally, keep learning. The API landscape evolves quickly, with new protocols and deployment models emerging. Stay informed about best practices and new tools. Share your experiences with the community—what worked, what didn't, and what surprised you. By sharing knowledge, we all improve our defenses.
Action plan for tomorrow: (1) Export your API gateway's endpoint list. (2) Set up a 24-hour network traffic capture. (3) Run a Git search for route definitions. (4) Merge the lists and identify the top 5 most exposed endpoints. (5) Decommission or secure those 5 within the next week. Repeat this cycle quarterly. Over time, your perimeter will be cleaner, your teams more disciplined, and your organization safer from the ghost forts that lurk in the shadows.
Comments (0)
Please sign in to post a comment.
Don't have an account? Create one
No comments yet. Be the first to comment!