{"id":262,"date":"2012-01-30T18:14:35","date_gmt":"2012-01-30T18:14:35","guid":{"rendered":"https:\/\/trouble.org\/?p=262"},"modified":"2012-01-30T21:01:41","modified_gmt":"2012-01-30T21:01:41","slug":"262","status":"publish","type":"post","link":"https:\/\/trouble.org\/?p=262","title":{"rendered":"Your most important systems are your least secure"},"content":{"rendered":"

I’ve a larger writing project afoot… putting it down to see how it looked in a different format.<\/em><\/p>\n

zen@trouble.org\/2012\/Draft<\/em><\/p>\n

Your most important systems are your least secure<\/strong><\/p>\n

Here\u2019s an easy one.\u00a0 Do you think that your most critical computers \u2013 the ones that are most important to your company\u2019s health, wealth, and well-being \u2013 are among your most secure?\u00a0 If you answered yes you\u2019re not alone in thinking so, but you\u2019re probably wrong.<\/p>\n

The roaring nineties saw the migration of wealth to the Internet \u2013 a land of seemingly endless opportunities \u2013 and spawned such capitalistic giants as Amazon, eBay, and Google.\u00a0 Venture capitalists poured hundreds of billions of dollars to fund innovative (and, in some cases, just plain crazy) ideas in an effort to profit from and tame this strange new landscape.\u00a0 Serious commercial and corporate computer security was still in its infancy, and the security product marketplace was just getting off the ground.<\/p>\n

Despite the money flowing into the ether very little was known about how good or bad security was in the wild.\u00a0 So in 1996 I took a sort of security health examination over a wide variety of high profile systems in order to assess their security.\u00a0 Running a modified version of SATAN (a network security scanner I had co-authored with Wietse Venema) I tried to examine the most important and interesting sites on the Internet at the time, including online banks, newspapers, credit unions, government institutions, and one of the earliest profitable net ventures: porn.<\/p>\n

It turned out that of the approximately 2,000 Internet facing servers I examined, nearly two thirds<\/strong> of them had significant security problems. About a third of them could have been broken into with almost no effort at all. \u00a0All in all about 3\/4 of all of these surveyed sites could have been compromised if significant force and effort had been applied[1]<\/a>.<\/p>\n

While bad in itself, what seemed even more curious was the fact that if you compared these important sites to a set of random sites I had also scanned as a scientific control, the random systems only had about half<\/em> the problems as their critical cousins!\u00a0 It might be worth noting that the porn sites were better defended than the banks \u2013 perhaps at the time that was were the real online money was \u2013 but c\u2019est la vie.<\/p>\n

While initially counter-intuitive, there were a variety of reasons the results held true.\u00a0 These busy and important computers simply did more, juggling web services, email, name service and other important data \u2013 keeping track of all of these metaphorical balls was a difficult task.\u00a0 Businesses and other organizations were coming to the net expecting turnkey solutions to their company needs that simply weren\u2019t available at the time, and it showed. Combine this with a general ignorance about what is important about security along with an emphasis on functionality and performance, you have a situation like \u2013 well, a situation that looks pretty similar to what we have now.<\/p>\n

In general the security of a computer degrades in proportion to the amount the system is used.\u00a0 Run a vulnerability scan against your average big server and a laptop, and the scanner will uncover fewer security issues on the laptop than the server.\u00a0 The laptop may seem to have paper-thin walls, but it is generally used to perform simple tasks and functions.\u00a0 Keeping all the different and complicated types of servers with a cornucopia of operating systems, applications, configurations, owners, processes and procedures secure is an extremely difficult task.\u00a0 Laptops, on the other hand, are mostly used as provisioned by IT, are relatively identical, and can generally be kept in a much more consistent and secure state with central management.<\/p>\n

Big iron, big problems<\/strong><\/p>\n

Fast-forwarding to today\u2019s world and this book\u2019s emphasis on internal networks, what does this mean? Internally \u2013 that is, behind your company\u2019s firewall or external network defenses \u2013 it\u2019s even worse.\u00a0 The odds are good that your SAP ERP, PeopleSoft, Oracle and other large applications are probably among the worst secured systems in your company.\u00a0\u00a0 But how could this be?\u00a0 After all, after spending tens of millions of dollars on the damn things[2]<\/a> why couldn\u2019t they have put a decent lock on the door?<\/p>\n

To start with these large environments are complex and use non-standard applications that leverage highly customized (remember that professional services bill?) programs that are uniquely tailored to your environment.\u00a0 Often conceived and created by 3rd<\/sup> parties, your internal security and design experts often had little or no input into the design or rollout of the application.\u00a0 And of course by now the original designers and system acolytes who really knew the system at the start have moved on or moved up, and they no longer have any operational duties or understanding of the current state.\u00a0 Secret backdoors that grant dangerous access could have been put in by the vendor (\u201cso we can get back in and fix things in case of emergency\u201d) or even by one of your own employees, and old accounts from bygone employees and support staff remain eternal because they\u2019re not linked into the same account provisioning system as the rest of your business.\u00a0 This means that when you think you\u2019ve disabled access to users, a variety of one-off accounts remain.\u00a0 And coordinating the communications and marching orders of all the teams involved \u2013 networking, operating systems, application, etc. \u2013 is a real challenge.\u00a0 Most of the maintenance is performed when reacting to operational issues that prevent the application from performing its duty \u2013 simply keeping it alive, not secure, is the order du jour.<\/p>\n

Often such applications are the culmination of several projects, and when the dust had settled years have passed before they were fully operational.\u00a0 Even if they used cutting edge designs and architecture when you started \u2013 and they weren\u2019t, because you don\u2019t use cutting edge technologies with critical systems \u2013 by the time the project is complete, with the usual budget and time overruns, you have a system that\u2019s nearly obsolete as it goes into production.\u00a0 All the change requests you made to add functionality or change the workflow have damaged the initial clean design and have added bloat that is ill understood and introduces uncertainty.\u00a0 And as the years go by, the optimistic original date given to your by the consultants is a distant memory, and they\u2019ve finally gotten it to work \u2013 well by then you try to ignore the duct tape that holds it all together and are happy it runs at all.\u00a0 Not to mention you can\u2019t replace it without spending many millions more, so you\u2019re stuck with these decade old dinosaurs that no one knows how to fix anymore and are often past end of life for the vendors that originally sold you the system (if the vendors themselves haven\u2019t been sold, gone out of business, or given up computers and gone into vacuum cleaner sales.)<\/p>\n

These initial details would be enough to sink the security of most \u2013 old, very complex, ill-understood, non-standard systems that are unsupported by the major vendors are a recipe for disaster.\u00a0 But unfortunately the business and operational problems are often a bit worse.<\/p>\n

A little knowledge\u2026 doesn\u2019t get you very far<\/strong><\/p>\n

So what do you think happens when the corporate security team finds a problem with one of these crucial applications (perhaps through a scheduled audit, network scan, or some other mechanism) and tells them there\u2019s a serious security problem that must be fixed?\u00a0 In a production environment the number one goal for these critical components is to have them work when you need them \u2013 e.g. have high uptime and availability.\u00a0 So naturally the application owners view anything that might jeopardize this ambition with deep suspicion, if not outright hostility.<\/p>\n

Owners resist change mightily, and might feel justified in asking reasonable questions like \u201ccan you guarantee that patch or change the system won\u2019t harm anything?\u201d\u00a0 Or perhaps the dreaded \u201cwhat\u2019s the business justification?\u201d\u00a0 Quotes along the lines of \u201cit costs the company $1,000 per second this is down!\u201d are common and somewhat reasonable pushbacks that security professionals encounter.<\/p>\n

And because of the great expense in setting up one of these monoliths, there are often either very limited or no test labs or QA environments that may be used for checking out any proposed changes, causing changes become even more dangerous when you can\u2019t study the impact in a risk-free environment. \u00a0Even if the repair is deemed critical (or at least worthwhile), it\u2019s going to take time \u2013 sometimes years \u2013 to repair issues in anything but extraordinary situations.\u00a0 And simply forget about fixing anything if the date is anywhere near the end of quarter, or the fiscal or calendar year, or \u2013 god forbid \u2013 if you\u2019ve missed the maintenance window (it\u2019s common for larger companies to have a freezes or slowdown windows of maintenance that consumes 25-50% of the calendar year or more.)<\/p>\n

Even with off-the-shelf solutions vendors will issue dire warnings or void your warranty (and their liability, as tenuous as that ever is.)\u00a0 This is especially true but not unique for health-related or mission critical systems: for instance SIEMENS, one of the largest manufacturers of industrial control systems, has very specific advice:<\/p>\n

A running process control plant should never be checked with penetration test tools! The use of penetration test tools is always associated with the risk of permanent damage to the tested system (or the installation or configuration of the system)<\/em>[3]<\/a><\/strong><\/em><\/p>\n

One might wonder if it\u2019s true that a scanner can cause permanent damage it might be a better idea of increasing the system defenses rather than not test it.\u00a0 Of course you don\u2019t want to damage something as critical as a Fizbit 970k that monitors the Gonzolar Zed\/2 that helps safeguard the plutonium in your reactor.\u00a0 But with the increased exposure of all systems to the Internet virus it becomes more and more vital to ensure that in real-world areas security remains bona fide.\u00a0 If a system is too dangerous to test, it might be too dangerous to run.<\/p>\n

But even if security can find issues and convince the owners to fix the problem, the windows of vulnerability that the problems exist are substantially longer than with more homogenous computers where patches and fixes can be rolled out in a more moderate length of time.\u00a0 All this might prompt you to ask \u2013 why bother to fix the problems, if it\u2019s so hard to address in these ancient systems?\u00a0\u00a0 Intruders break into systems by exploiting commonly known or easy to exploit with a punch of a button.\u00a0 At some point in time you might ask \u2013 are all those potential buttons that you let stay on your critical systems worth fixing or not?<\/p>\n

Of course getting consensus on what is really important and deserving of protection is not an easy task.\u00a0 But that pales in comparison as to tracking down who is responsible for these business assets and who is in charge of ensuring the various components that make up the systems that run the applications or storing key assets are in good shape.\u00a0 We can thank Google and their contributions on making search such a tremendous tool that it\u2019s easy to forget what a hard problem it was.\u00a0 However, the differences in the nature of information on the Internet \u2013 where people most put it out to be found \u2013 and data that lives inside a large corporation where most of the time business owners only want a small fragment of data to be discovered (e.g. their business unit landing page) is large.\u00a0 \u00a0And this doesn\u2019t even begin to touch upon the difficulties that multinational companies have, nor with the near constant M&A\u2019s that larger companies seem to thrive upon.\u00a0 How well do you know what goes on inside of your new acquisitions?\u00a0 I recall being at a company that couldn\u2019t even install security software on its European branches because of EU privacy concerns.<\/p>\n

On one hand it might seem striking that there are such disconnects, but in larger organizations the insular nature of these systems, which are generally out of the direct involvement of central IT, owned by a business unit that doesn\u2019t have the resources or knowledge on how to run complex systems, let alone how to integrate them into the processes and standards of rest of their company.\u00a0\u00a0 All this conspires to ensure that your crown jewels often rest on systems that are run in an ad hoc, insecure fashion that are out of sight of security and the rest of the business.<\/p>\n

But even assuming the security or audit teams know of the existence of a given application, actually discovering security weaknesses is yet another adventure.\u00a0 It\u2019s pretty much a given that any computer that is not well-managed and without frequent security updates and appropriate security controls in place will give numerous errors when scanned by vulnerability scanners, configuration auditors, denial of service testers, application probers, and the like.\u00a0 Many of these tools are tried and true arrows in the security tools quiver; while aggressive, and certainly not perfect, they can provide hard data on how bad things are for a given application.\u00a0 But turning on the heat and running these aggressive products against even relatively stable computers is dangerous \u2013 this is because by design security probes steal or starve resources, impact performance, crash programs or take entire systems down routinely.\u00a0 Worse still application scanners can inject bad or random data into your systems and databases as part of a testing.\u00a0 These injections of data can potentially damage, corrupt, or otherwise compromising the integrity of your data.\u00a0 None of these side effects will make the owner happy, who on good days doesn\u2019t want to look at their application too closely lest it keel over.<\/p>\n

As a result many companies simply disallow the really heavy scanning or security testing of critical assets.\u00a0 While understandable \u2013 you don\u2019t want your ERP system capsizing \u2013 do you really think that the bad folks will set their phasers on stun?\u00a0 It\u2019s going to be an all-out assault, set to kill.\u00a0 And they will.<\/p>\n

In the end it\u2019s a simple case of dueling metrics \u2013 do you keep the systems purring and happy, or introduce business risks by finding and fixing the problems?\u00a0\u00a0 The rallying cry of \u201cit\u2019s never been a problem before!\u201d rings through the halls of business as a talisman against change.\u00a0\u00a0 And as a result there is an inverse correlation between importance of systems and security.<\/p>\n

Give up Hope?<\/strong><\/p>\n

I\u2019m sure most of us would just love security to go away.\u00a0 But it isn\u2019t, and it\u2019s growing in import as our most crucial assets, services, and information are entrenched in the virtual world.\u00a0 And to be fair, some crucial applications are pretty well secured by nearly any measure. \u00a0As for the rest, all is not lost.\u00a0 With a bit (sometimes a large bit!) of effort even the worst offenders can become productive and<\/strong> secure members of society. \u00a0It\u2019s easy to say high-level statements such as one should ensure that IT and security should be serving the business\u2019 needs while leveraging a good governance framework and ensuring that resources and efforts be directed at high business value assets that are at risk.\u00a0\u00a0 But changing how IT and security works isn\u2019t something that can change overnight \u2013 or, if it could, you wouldn\u2019t like the results.\u00a0 It\u2019s more like steering an oil supertanker, you have to plan a fair bit in advance and wait for the results.<\/p>\n

Paralysis is an option with so many choices to be made.\u00a0 The common consensus is to use one or more of the popular large frameworks to guide your operations.\u00a0 ITIL, COBIT, ISO-27001, and others aren\u2019t a panacea but are a nice structure to hang your proverbial hat on.\u00a0\u00a0 Without the proper processes and business practices in place to respond to changes to the risk and threat landscape, you\u2019ll quickly end up exactly where you started.\u00a0 The goal should be to not let the business owners mandate their own security \u2013 the holy trinity of business, process, and technology have to be in congruence to the strategic aims of the rest of the organization.\u00a0 Treating these mission critical assets as nothing unique or strange, as well as to ensure that they fit into your normal processes and best practices, are the key.\u00a0 Because of how they\u2019re put together they naturally they won\u2019t be identical to other servers and hosts within your organization, but they absolutely must be well understood, documented, and integrated within your business and process framework.<\/p>\n

Of course in extreme cases the owners might claim that any changes at all will sink the application in question.\u00a0 If it really is this brittle you have little choice but to start thinking how to replace either the owners or the application itself; having a critical resource that could be destroyed by change is a serious risk to your business and should be addressed before disaster hits.<\/p>\n

If at all possible your best architects and security professionals should be engaged as part of your normal design process \u2013 let them help work alongside the business and 3rd<\/sup> party teams to create a set of resilient, redundant, and mature processes and technologies that will dovetail and amplify the business goals and success of the application.\u00a0 \u00a0Have security not only baked into the design from the start but integrating security into the ongoing monitoring and maintenance processes is vital.<\/p>\n

To be sure, use suppliers and vendors with long-standing history of stability and reliability, and require that they fully document any work so as time goes on this knowledge won\u2019t be lost (but try to engage in dialogue and talk some sense into suppliers like SIEMENS who don\u2019t want you to even look at their systems!) \u00a0If at all possible segregate mission critical applications either virtually or physically from the rest of your environment; this will prevent security incidents in lesser systems from metastasizing to the big guns \u2013 but don\u2019t lose track of them!\u00a0 Always document the purpose of such large systems and catalogue such basics as who the business owner is, how the security is handled as well as the controls that are in place and how to monitor their health and wellbeing.<\/p>\n

Above all never let fear or uncertainty drive your actions \u2013 integrate an appropriate understanding of business goals into the equation and make decisions based on an ongoing survey of the processes and risks involved.\u00a0 The results might cost you money \u2013 but paralysis or indecision can cost much more.<\/p>\n

 <\/p>\n

\n
\n
\n

[1]<\/a> I personally didn\u2019t compromise any systems, just essentially ran some popular and very well-known security diagnostic tests, so the numbers were fairly conservative; any attackers would presumably be more aggressive. Large-scale non-intrusive surveys by others have had surprisingly similar numbers no matter what methods used.<\/p>\n<\/div>\n

\n

[2]<\/a> Even moderately sized companies (say, 1000 people or more) routinely spend millions on large and important projects.\u00a0 The really expensive projects, such as an ERP implementation, easily can run orders of magnitude more.\u00a0 Large enterprise upgrade or retrofitting programs routinely exceed one hundred million or more in hardware, software, services, and upgrade costs, while history is full of multi-billion dollar IT project failures.<\/p>\n<\/div>\n

\n

[3]<\/a> From the SIEMENS security whitepaper \u201cSecurity concept PCS 7 and WinCC<\/a>\u201d.<\/p>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

I’ve a larger writing project afoot… putting it down to see how it looked in a different format. zen@trouble.org\/2012\/Draft Your most important systems are your least secure Here\u2019s an easy one.\u00a0 Do you think that your most critical computers \u2013 the ones that are most important to your company\u2019s health, wealth, and well-being \u2013 are […]<\/p>\n","protected":false},"author":44,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[19,4],"tags":[],"class_list":["post-262","post","type-post","status-publish","format-standard","hentry","category-philosophy","category-security"],"_links":{"self":[{"href":"https:\/\/trouble.org\/index.php?rest_route=\/wp\/v2\/posts\/262","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/trouble.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/trouble.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/trouble.org\/index.php?rest_route=\/wp\/v2\/users\/44"}],"replies":[{"embeddable":true,"href":"https:\/\/trouble.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=262"}],"version-history":[{"count":7,"href":"https:\/\/trouble.org\/index.php?rest_route=\/wp\/v2\/posts\/262\/revisions"}],"predecessor-version":[{"id":269,"href":"https:\/\/trouble.org\/index.php?rest_route=\/wp\/v2\/posts\/262\/revisions\/269"}],"wp:attachment":[{"href":"https:\/\/trouble.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=262"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/trouble.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=262"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/trouble.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=262"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}