Current Operational Security Practices in Internet Service Provider Environments

Device physical access pertains to protecting the physical location and access of the layer 2 or layer 3 network infrastructure device. Physical security is a large field of study/practice in and of itself, arguably the largest, oldest, and most well-understood area of security. Although it is important to have contingency plans for natural disasters, such as earthquakes and floods, which can cause damage to networking devices, this is out of the scope of this document. Here, we concern ourselves with protecting access to the physical location and how a device can be further protected from unauthorized access if the physical location has been compromised, i.e., protecting the console access. This is aimed largely at stopping an intruder with physical access from gaining operational control of the device(s). Note that nothing will stop an attacker with physical access from effecting a denial-of-service attack, which can be easily accomplished by powering off the device or just unplugging some cables.

If any intruder gets physical access to a layer 2 or layer 3 device, the entire network infrastructure can be under the control of the intruder. At a minimum, the intruder can take the compromised device out of service, causing network disruption, the extent of which depends on the network topology. A worse scenario is where the intruder uses this device to crack the console password, gaining complete control of the device (perhaps without anyone detecting such a compromise, or to attach another network device onto a port and siphon off data with which the intruder can ascertain the network topology) and the entire network. The threat of gaining physical access can be realized in a variety of ways, even if critical devices are under high security. Cases still occur where attackers have impersonated maintenance workers to gain physical access to critical devices that have caused major outages and privacy compromises. Insider attacks from authorized personnel also pose a real threat and must be adequately recognized and addressed.

For physical device security, equipment is kept in highly restrictive environments. Only authorized users with card-key badges have access to any of the physical locations that contain critical network infrastructure devices. These card-key systems keep track of who accessed which location and at what time. Most cardkey systems have a fail-back "master key" in case the card system is down. This "master key" usually has limited access and its use is also carefully logged (which should only happen if the card-key system is NOT online/functional). All console access is always password protected and the login time is set to time out after a specified amount of inactivity - typically between 3-10 minutes. The type of privileges that you obtain from a console login varies between separate vendor devices. In some cases you get initial basic access and need to perform a second authentication step to get more privileged access (i.e., enable or root). In other vendors, you get the more privileged access when you log into the console as root, without requiring a second authentication step. How ISPs manage these logins vary greatly, although many of the larger ISPs employ some sort of AAA mechanism to help automate privilege-level authorization and utilize the automation to bypass the need for a second authentication step. Also, many ISPs define separate classes of users to have different privileges while logged onto the console. Typically, all console access is provided via an out-of-band (OOB) management infrastructure, which is discussed in Section 2.2 of this document.

The following security services are offered through the use of the practices described in the previous section: User Authentication - All individuals who have access to the physical facility are authenticated. Console access is authenticated. User Authorization - An authenticated individual has implicit authorization to perform commands on the device. In some cases, multiple authentication is required to differentiate between basic and more privileged access. Data Origin Authentication - Not applicable. Access Control - Not applicable. Data Integrity - Not applicable. Data Confidentiality - Not applicable. Auditing/Logging - All access to the physical locations of the infrastructure equipment is logged via electronic card-key systems. All console access is logged (refer to Section 2.2 of this document for more details). DoS Mitigation - Not applicable.

Physical security is relevant to operational security practices as described in this document, mostly from a console-access perspective. Most ISPs provide console access via an OOB management infrastructure, which is discussed in Section 2.2 of this document. The physical and logical authentication and logging systems should be run independently of each other and should reside in different physical locations. These systems need to be secured to ensure that they themselves will not be compromised, which could give the intruder valuable authentication and logging information. Social engineering plays a big role in many physical access compromises. Most ISPs have set up training classes and awareness programs to educate company personnel to deny physical access to people who are not properly authenticated or authorized to have physical access to critical infrastructure devices.

In-band management is generally considered to be device access , where the control traffic takes the same data path as the data that traverses the network. Out-of-band management is generally considered to be device access , where the control traffic takes a separate path as the data that traverses the network. In many environments, device management for layer 2 and layer 3 infrastructure devices is deployed as part of an out-of-band management infrastructure, although there are some instances where it is deployed in-band as well. Note that while many of the security concerns and practices are the same for OOB management and in-band management, most ISPs prefer an OOB management system, since access to the devices that make up this management network are more vigilantly protected and considered to be less susceptible to malicious activity. Console access is always architected via an OOB network. Presently, the mechanisms used for either in-band management or OOB are via virtual terminal access (i.e., Telnet or SSH), Simple Network Management Protocol (SNMP), or HTTP. In all large ISPs that were interviewed, HTTP management was never used and was explicitly disabled. Note that file transfer protocols (TFTP, FTP, and SCP) will be covered in Section 2.5 of this document.

For device management, passive attacks are possible if someone has the capability to intercept data between the management device and the managed device. The threat is possible if a single infrastructure device is somehow compromised and can act as a network sniffer, or if it is possible to insert a new device that acts as a network sniffer. Active attacks are possible for both on-path and off-path scenarios. For on-path active attacks, the situation is the same as for a passive attack, where either a device has to already be compromised or a device can be inserted into the path. For off-path active attacks, where a topology subversion is required to reroute traffic and essentially bring the attacker on-path, the attack is generally limited to message insertion or modification.

Confidentiality violations can occur when a miscreant intercepts any management data that has been sent in cleartext or with weak encryption. This includes interception of usernames and passwords with which an intruder can obtain unauthorized access to network devices. It can also include other information, such as logging or configuration information, if an administrator is remotely viewing local logfiles or configuration information.

If username/password information was encrypted but the cryptographic mechanism used made it easy to capture data and break the encryption key, the device management traffic could be compromised. The traffic would need to be captured either by eavesdropping on the network or by being able to divert traffic to a malicious user.

For a replay attack to be successful, the management traffic would need to first be captured either on-path or diverted to an attacker to later be replayed to the intended recipient.

Data can be manipulated by someone in control of intermediary hosts. Forging data is also possible with IP spoofing, where a remote host sends out packets that appear to come from another, trusted host.

A man-in-the-middle attack attacks the identity of a communicating peer rather than the data stream itself. The attacker intercepts traffic that is sent from a management system to the networking infrastructure device and traffic that is sent from the network infrastructure device to the management system.

OOB management is done via a terminal server at each location. SSH access is used to get to the terminal server from where sessions to the devices are initiated. Dial-in access is deployed as a backup if the network is not available. However, it is common to use dial-back, encrypting modems, and/or one-time-password (OTP) modems to avoid the security weaknesses of plain dial-in access. All in-band management and OOB management access to layer 2 and layer 3 devices is authenticated. The user authentication and authorization is typically controlled by an AAA server (i.e., Remote Authentication Dial-in User Service (RADIUS) and/or Terminal Access Controller Access-Control System (TACACS+)). Credentials used to determine the identity of the user vary from static username/password to one-time username/password schemes such as Secure-ID. Static username/passwords are expired after a specified period of time, usually 30 days. Every authenticated entity via AAA is an individual user for greater granularity of control. Note that often the AAA server used for OOB management authentication is a separate physical device from the AAA server used for in-band management user authentication. In some deployments, the AAA servers used for device management authentication/authorization/accounting are on separate networks to provide a demarcation for any other authentication functions. For backup purposes, there is often a single local database entry for authentication that is known to a very limited set of key personnel. It is usually the highest privilege-level username/password combination, which in most cases is the same across all devices. This local device password is routinely regenerated once every 2-3 months, and is also regenerated immediately after an employee who had access to that password leaves the company or is no longer authorized to have knowledge of that password. Each individual user in the AAA database is configured with specific authorization capability. Specific commands are either individually denied or permitted, depending on the capability of the device to be accessed. Multiple privilege levels are deployed. Most individuals are authorized with basic authorization to perform a minimal set of commands, while a subset of individuals are authorized to perform more privileged commands. Securing the AAA server is imperative and access to the AAA server itself is strictly controlled. When an individual leaves the company, his/her AAA account is immediately deleted and the TACACS/RADIUS shared secret is reset for all devices. Some management functions are performed using command line interface (CLI) scripting. In these scenarios, a dedicated user is used for the identity in scripts that perform CLI scripting. Once authenticated, these scripts control which commands are legitimate, depending on authorization rights of the authenticated individual. SSH is always used for virtual terminal access to provide for an encrypted communication channel. There are exceptions due to equipment limitations which are described in the additional considerations section. If SNMP is used for management, it is for read queries only and restricted to specific hosts. If possible, the view is also restricted to only send the information that the management station needs, rather than expose the entire configuration file with the read-only SNMP community. The community strings are carefully chosen to be difficult to crack and there are procedures in place to change these community strings between 30-90 days. If systems support two SNMP community strings, the old string is replaced by first configuring a second, newer community string and then migrating over from the currently used string to the newer one. Most large ISPs have multiple SNMP systems accessing their routers so it takes more then one maintenance period to get all the strings fixed in all the right systems. SNMP RW is not used and is disabled by configuration. Access control is strictly enforced for infrastructure devices by using stringent filtering rules. A limited set of IP addresses are allowed to initiate connections to the infrastructure devices and are specific to the services to which they are to limited (i.e., SSH and SNMP). All device management access is audited and any violations trigger alarms that initiate automated email, pager, and/or telephone notifications. AAA servers keep track of the authenticated entity as well as all the commands that were carried out on a specific device. Additionally, the device itself logs any access control violations (i.e., if an SSH request comes in from an IP address that is not explicitly permitted, that event is logged so that the offending IP address can be tracked down and investigations made as to why it was trying to access a particular infrastructure device)

The security services offered for device OOB management are nearly identical to those of device in-band management. Due to the critical nature of controlling and limiting device access, many ISPs feel that physically separating the management traffic from the normal customer data traffic will provide an added level of risk mitigation and limit the potential attack vectors. The following security services are offered through the use of the practices described in the previous section: User Authentication - All individuals are authenticated via AAA services. User Authorization - All individuals are authorized via AAA services to perform specific operations once successfully authenticated. Data Origin Authentication - Management traffic is strictly filtered to allow only specific IP addresses to have access to the infrastructure devices. This does not alleviate risk the from spoofed traffic, although when combined with edge filtering using BCP38 and BCP84 guidelines (discussed in Section 2.5), then the risk of spoofing is mitigated, barring a compromised internal system. Also, using SSH for device access ensures that no one can spoof the traffic during the SSH session. Access Control - Management traffic is filtered to allow only specific IP addresses to have access to the infrastructure devices. Data Integrity - Using SSH provides data integrity and ensures that no one has altered the management data in transit. Data Confidentiality - Using SSH provides data confidentiality. Auditing/Logging - Using AAA provides an audit trail for who accessed which device and which operations were performed. DoS Mitigation - Using packet filters to allow only specific IP addresses to have access to the infrastructure devices. This limits but does not prevent spoofed DoS attacks directed at an infrastructure device. However, the risk is lowered by using a separate physical network for management purposes.

Password selection for any device management protocol used is critical to ensure that the passwords are hard to guess or break using a brute-force attack. IP security (IPsec) is considered too difficult to deploy, and the common protocol to provide for confidential management access is SSH. There are exceptions for using SSH due to equipment limitations since SSH may not be supported on legacy equipment. In some cases, changing the host name of a device requires an SSH rekey event since the key is based on some combination of host name, Message Authentication Code (MAC) address, and time. Also, in the case where the SSH key is stored on a route processor card, a re-keying of SSH would be required whenever the route processor card needs to be swapped. Some providers feel that this operational impact exceeds the security necessary and instead use Telnet from trusted inside hosts (called 'jumphosts' or 'bastion hosts') to manage those devices. An individual would first SSH to the jumphost and then Telnet from the jumphost to the actual infrastructure device, fully understanding that any passwords will be sent in the clear between the jumphost and the device to which it is connecting. All authentication and authorization is still carried out using AAA servers. In instances where Telnet access is used, the logs on the AAA servers are more verbose and more attention is paid to them to detect any abnormal behavior. The jumphosts themselves are carefully controlled machines and usually have limited access. Note that Telnet is NEVER allowed to an infrastructure device except from specific jumphosts; i.e., packet filters are used at the console server and/or infrastructure device to ensure that Telnet is only allowed from specific IP addresses. With thousands of devices to manage, some ISPs have created automated mechanisms to authenticate to devices. As an example, Kerberos has been used to automate the authentication process for devices that have support for Kerberos. An individual would first log in to a Kerberized UNIX server using SSH and generate a Kerberos 'ticket'. This 'ticket' is generally set to have a lifespan of 10 hours and is used to automatically authenticate the individual to the infrastructure devices. In instances where SNMP is used, some legacy devices only support SNMPv1, which then requires the provider to mandate its use across all infrastructure devices for operational simplicity. SNMPv2 is primarily deployed since it is easier to set up than v3.

This section refers to how traffic is handled that traverses the network infrastructure device. The primary goal of ISPs is to forward customer traffic. However, due to the large amount of malicious traffic that can cause DoS attacks and render the network unavailable, specific measures are sometimes deployed to ensure the availability to forward legitimate customer traffic.

Any data traffic can potentially be attack traffic and the challenge is to detect and potentially stop forwarding any of the malicious traffic. The deliberately sourced attack traffic can consist of packets with spoofed source and/or destination addresses or any other malformed packet that mangle any portion of a header field to cause protocol-related security issues (such as resetting connections, causing unwelcome ICMP redirects, creating unwelcome IP options, or packet fragmentations).

Filtering and rate limiting are the primary mechanism to provide risk mitigation of malicious traffic rendering the ISP services unavailable. However, filtering and rate limiting of data path traffic is deployed in a variety of ways, depending on how automated the process is and what the capabilities and performance limitations of the existing deployed hardware are. The ISPs that do not have performance issues with their equipment follow BCP38 and BCP84 guidelines for ingress filtering. BCP38 recommends filtering ingress packets with obviously spoofed and/or 'reserved' source addresses to limit the effects of denial-of-service attacks, while BCP84 extends the recommendation for multi-homed environments. Filters are also used to help alleviate issues between service providers. Without any filtering, an inter-exchange peer could steal transit just by using static routes, and essentially redirect data traffic. Therefore, some ISPs have implemented ingress/egress filters that block unexpected source and destination addresses not defined in the above-mentioned documents. Null routes and black-hole triggered routing are used to deter any detected malicious traffic streams. These two techniques are described in more detail in Section 2.8 below. Most ISPs consider layer 4 filtering useful, but it is only implemented if performance limitations allow for it. Since it poses a large administrative overhead and ISPs are very much opposed to acting as the Internet firewall, Layer 4 filtering is typically implemented as a last option. Netflow is used for tracking traffic flows, but there is some concern whether sampling is good enough to detect malicious behavior. Unicast Reverse Path Forwarding (RPF) is not consistently implemented. Some ISPs are in the process of doing so, while other ISPs think that the perceived benefit of knowing that spoofed traffic comes from legitimate addresses are not worth the operational complexity. Some providers have a policy of implementing uRPF at link speeds of Digital Signal 3 (DS3) and below, which was due to the fact that all hardware in the network supported uRPF for DS3 speeds and below. At higher-speed links, the uRPF support was inconsistent and it was easier for operational people to implement a consistent solution.

User Authentication - Not applicable. User Authorization - Not applicable. Data Origin Authentication - When IP address filtering per BCP38, BCP84, and uRPF are deployed at network edges it can ensure that any spoofed traffic comes from at least a legitimate IP address and can be tracked. Access Control - IP address filtering and layer 4 filtering is used to deny forbidden protocols and limit traffic destined for infrastructure device itself. Filters are also used to block unexpected source/destination addresses. Data Integrity - Not applicable. Data Confidentiality - Not applicable. Auditing/Logging - Filtering exceptions are logged for potential attack traffic. DoS Mitigation - Black-hole triggered filtering and rate-limiting are used to limit the risk of DoS attacks.

For layer 2 devices, MAC address filtering and authentication is not used in large-scale deployments. This is due to the problems it can cause when troubleshooting networking issues. Port security becomes unmanageable at a large scale where thousands of switches are deployed. Rate limiting is used by some ISPs, although other ISPs believe it is not really useful, since attackers are not well-behaved and it doesn't provide any operational benefit over the complexity. Some ISPs feel that rate limiting can also make an attacker's job easier by requiring the attacker to send less traffic to starve legitimate traffic that is part of a rate limiting scheme. Rate limiting may be improved by developing flow-based rate-limiting capabilities with filtering hooks. This would improve the performance as well as the granularity over current capabilities. Lack of consistency regarding the ability to filter, especially with respect to performance issues, cause some ISPs not to implement BCP38 and BCP84 guidelines for ingress filtering. One such example is at edge boxes, where up to 1000 T1s connecting into a router with an OC-12 (Optical Carrier) uplink. Some deployed devices experience a large performance impact with filtering, which is unacceptable for passing customer traffic through, though ingress filtering (uRPF) might be applicable at the devices that are connecting these aggregation routers. Where performance is not an issue, the ISPs make a tradeoff between management versus risk.

The routing control plane deals with all the traffic that is part of establishing and maintaining routing protocol information.

Attacks on the routing control plane can be from both passive or active sources. Passive attacks are possible if someone has the capability to intercept data between the communicating routing peers. This can be accomplished if a single routing peer is somehow compromised and can act as a network sniffer, or if it is possible to insert a new device that acts as a network sniffer. Active attacks are possible for both on-path and off-path scenarios. For on-path active attacks, the situation is the same as for a passive attack, where either a device has to already be compromised or a device can be inserted into the path. This may lead to an attacker impersonating a legitimate routing peer and exchanging routing information. Unintentional active attacks are more common due to configuration errors, which cause legitimate routing peers to feed invalid routing information to other neighboring peers. For off-path active attacks, the attacks are generally limited to message insertion or modification, which can divert traffic to illegitimate destinations, causing traffic to never reach its intended destination.

Confidentiality violations can occur when a miscreant intercepts any of the routing update traffic. This is becoming more of a concern because many ISPs are classifying addressing schemes and network topologies as private and proprietary information. It is also a concern because the routing protocol packets contain information that may show ways in which routing sessions could be spoofed or hijacked. This in turn could lead into a man-in-the-middle attack, where the miscreants can insert themselves into the traffic path or divert the traffic path and violate the confidentiality of user data.

If any cryptographic mechanism was used to provide for data integrity and confidentiality, an offline cryptographic attack could potentially compromise the data. The traffic would need to be captured either by eavesdropping on the network or by being able to divert traffic to a malicious user. Note that by using cryptographically protected routing information, the latter would require the cryptographic key to already be compromised anyway, so this attack is only feasible if a device was able to eavesdrop and capture the cryptographically protected routing information.

For a replay attack to be successful, the routing control plane traffic would need to first be captured either on-path or diverted to an attacker to later be replayed to the intended recipient. Additionally, since many of these protocols include replay protection mechanisms, these would also need to be subverted, if applicable.

Routing control plane traffic can be manipulated by someone in control of intermediate hosts. In addition, traffic can be injected by forging IP addresses, where a remote router sends out packets that appear to come from another, trusted router. If enough traffic is injected to be processed by limited memory routers, it can cause a DoS attack.

A man-in-the-middle attack attacks the identity of a communicating peer rather than the data stream itself. The attacker intercepts traffic that is sent from one routing peer to the other and communicates on behalf of one of the peers. This can lead to a diversion of the user traffic to either an unauthorized receiving party or cause legitimate traffic to never reach its intended destination.

Securing the routing control plane takes many features, which are generally deployed as a system. Message Digest 5 (MD5) authentication is used by some ISPs to validate the sending peer and to ensure that the data in transit has not been altered. Some ISPs only deploy MD5 authentication at the customers' request. Additional sanity checks to ensure with reasonable certainty that the received routing update was originated by a valid routing peer include route filters and the Generalized TTL Security Mechanism (GTSM) feature (sometimes also referred to as the TTL-Hack). The GTSM feature is used for protocols such as the Border Gateway Protocol (BGP), and makes use of a packet's Time To Live (TTL) field (IPv4) or Hop Limit (IPv6) to protect communicating peers. If GTSM is used, it is typically deployed only in limited scenarios between internal BGP peers due to lack of consistent support between vendor products and operating system versions. Packet filters are used to limit which systems can appear as a valid peer, while route filters are used to limit which routes are believed to be from a valid peer. In the case of BGP routing, a variety of policies are deployed to limit the propagation of invalid routing information. These include: incoming and outgoing prefix filters for BGP customers, incoming and outgoing prefix filters for peers and upstream neighbors, incoming AS-PATH filter for BGP customers, outgoing AS-PATH filter towards peers and upstream neighbors, route dampening and rejecting selected attributes and communities. Consistency between these policies varies greatly and there is a definite distinction whether the other end is an end-site vs an internal peer vs another big ISP or customer. Mostly ISPs do prefix-filter their end-site customers, but due to the operational constraints of maintaining large prefix filter lists, many ISPs are starting to depend on BGP AS-PATH filters to/from their peers and upstream neighbors. In cases where prefix lists are not used, operators often define a maximum prefix limit per peer to prevent misconfiguration (e.g., unintentional de-aggregation or neighbor routing policy mis-configuration) or overload attacks. ISPs need to coordinate with each other what the expected prefix exchange is, and increase this number by some sane amount. It is important for ISPs to pad the max-prefix number enough to allow for valid swings in routing announcements, preventing an unintentional shut down of the BGP session. Individual implementation amongst ISPs are unique, and depending on equipment supplier(s), different implementation options are available. Most equipment vendors offer implementation options ranging from just logging excessive prefixes being received, to automatically shutting down the session. If the option of reestablishing a session after some pre-configured idle timeout has been reached is available, it should be understood that automatically reestablishing the session may potentially introduce instability continuously into the overall routing table if a policy mis-configuration on the adjacent neighbor is causing the condition. If a serious mis-configuration on a peering neighbor has occurred, then automatically shutting down the session and leaving it shut down until being manually cleared, is sometimes best and allows for operator intervention to correct as needed. Some large ISPs require that routes be registered in an Internet Routing Registry (IRR), which can then be part of the Routing Assets Database (RADb) - a public registry of routing information for networks in the Internet that can be used to generate filter lists. Some ISPs, especially in Europe, require registered routes before agreeing to become an eBGP peer with someone. Many ISPs also do not propagate interface IP addresses to further reduce attack vectors on routers and connected customers.

User Authentication - Not applicable. User Authorization - Not applicable. Data Origin Authentication - By using MD5 authentication and/or the TTL-hack, a routing peer can be reasonably certain that traffic originated from a valid peer. Access Control - Route filters, AS-PATH filters, and prefix limits are used to control access to specific parts of the network. Data Integrity - By using MD5 authentication, a peer can be reasonably certain that the data has not been modified in transit, but there is no mechanism to prove the validity of the routing information itself. Data Confidentiality - Not implemented. Auditing / Logging - Filter exceptions are logged. DoS Mitigation - Many DoS attacks are mitigated using a combination of techniques including: MD5 authentication, the GTSM feature, filtering routing advertisements to bogons, and filtering routing advertisements to one's own network.

So far the primary concern to secure the routing control plane has been to validate the sending peer and to ensure that the data in transit has not been altered. Although MD5 routing protocol extensions have been implemented, which can provide both services, they are not consistently deployed amongst ISPs. Two major deployment concerns have been implementation issues, where both software bugs and the lack of graceful re-keying options have caused significant network down times. Also, some ISPs express concern that deploying MD5 authentication will itself be a worse DoS attack victim and prefer to use a combination of other risk mitigation mechanisms such as GTSM (for BGP) and route filters. An issue with GTSM is that it is not supported on all devices across different vendors' products. IPsec is not deployed since the operational management aspects of ensuring interoperability and reliable configurations is too complex and time consuming to be operationally viable. There is also limited concern to the confidentiality of the routing information. The integrity and validity of the updates are of much greater concern. There is concern for manual or automated actions, which introduce new routes and can affect the entire routing domain.

Software upgrades and configuration changes are usually performed as part of either in-band or OOB management functions. However, there are additional considerations to be taken into account, which are enumerated in this section.

Attacks performed on system software and configurations can be both from passive or active sources. Passive attacks are possible if someone has the capability to intercept data between the network infrastructure device and the system which is downloading or uploading the software or configuration information. This can be accomplished if a single infrastructure device is somehow compromised and can act as a network sniffer, or if it is possible to insert a new device that acts as a network sniffer. Active attacks are possible for both on-path and off-path scenarios. For on-path active attacks, the situation is the same as for a passive attack, where either a device has to already be compromised or a device can be inserted into the path. For off-path active attacks, the attacks are generally limited to message insertion or modification where the attacker may wish to load illegal software or configuration files to an infrastructure device. Note that similar issues are relevant when software updates are downloaded from a vendor site to an ISPs network management system that is responsible for software updates and/or configuration information.

Confidentiality violations can occur when a miscreant intercepts any of the software image or configuration information. The software image may give an indication of exploits which the device is vulnerable to while the configuration information can inadvertently lead attackers to identify critical infrastructure IP addresses and passwords.

For a replay attack to be successful, the software image or configuration file would need to first be captured either on-path or diverted to an attacker to later be replayed to the intended recipient. Additionally, since many protocols do have replay protection capabilities, these would have to be subverted as well in applicable situations.

Software images and configuration files can be manipulated by someone in control of intermediate hosts. By forging an IP address and impersonating a valid host which can download software images or configuration files, invalid files can be downloaded to an infrastructure device. This can also be the case from trusted vendors who may unbeknownst to them have compromised trusted hosts. An invalid software image or configuration file can cause a device to hang and become inoperable. Spoofed configuration files can be hard to detect, especially when the only added command is to allow a miscreant access to that device by entering a filter allowing a specific host access and configuring a local username/password database entry for authentication to that device.

A man-in-the-middle attack attacks the identity of a communicating peer rather than the data stream itself. The attacker intercepts traffic that is sent between the infrastructure device and the host used to upload/download the system image or configuration file. He/she can then act on behalf of one or both of these systems. If an attacker obtained a copy of the software image being deployed, he could potentially exploit a known vulnerability and gain access to the system. From a captured configuration file, he could obtain confidential network topology information, or even more damaging information, if any of the passwords in the configuration file were not encrypted.

Images and configurations are stored on specific hosts that have limited access. All access and activity relating to these hosts are authenticated and logged via AAA services. When uploaded/downloading any system software or configuration files, either TFTP, FTP, or SCP can be used. Where possible, SCP is used to secure the data transfer and FTP is generally never used. All SCP access is username/password authenticated but since this requires an interactive shell, most ISPs will use shared key authentication to avoid the interactive shell. While TFTP access does not have any security measures, it is still widely used, especially in OOB management scenarios. Some ISPs implement IP-based restriction on the TFTP server, while some custom written TFTP servers will support MAC-based authentication. The MAC-based authentication is more common when using TFTP to bootstrap routers remotely. In most environments, scripts are used for maintaining the images and configurations of a large number of routers. To ensure the integrity of the configurations, every hour the configuration files are polled and compared to the previously polled version to find discrepancies. In at least one environment these, tools are Kerberized to take advantage of automated authentication (not confidentiality). 'Rancid' is one popular publicly available tool for detecting configuration and system changes. Filters are used to limit access to uploading/downloading configuration files and system images to specific IP addresses and protocols. The software images perform Cyclic Redundancy Checks (CRC) and the system binaries use the MD5 algorithm to validate integrity. Many ISPs expressed interest in having software image integrity validation based on the MD5 algorithm for enhanced security. In all configuration files, most passwords are stored in an encrypted format. Note that the encryption techniques used in varying products can vary and that some weaker encryption schemes may be subject to off-line dictionary attacks. This includes passwords for user authentication, MD5-authentication shared secrets, AAA server shared secrets, NTP shared secrets, etc. For older software that may not support this functionality, configuration files may contain some passwords in readable format. Most ISPs mitigate any risk of password compromise by either storing these configuration files without the password lines or by requiring authenticated and authorized access to the configuration files that are stored on protected OOB management devices. Automated security validation is performed on infrastructure devices using Network Mapping (Nmap) and Nessus to ensure valid configuration against many of the well-known attacks.

User Authentication - All users are authenticated before being able to download/upload any system images or configuration files. User Authorization - All authenticated users are granted specific privileges to download or upload system images and/or configuration files. Data Origin Authentication - Filters are used to limit access to uploading/downloading configuration files and system images to specific IP addresses. Access Control - Filters are used to limit access to uploading/downloading configuration files and system images to specific IP addresses and protocols. Data Integrity - All systems use either a CRC-check or MD5 authentication to ensure data integrity. Also, tools such as rancid are used to automatically detect configuration changes. Data Confidentiality - If the SCP protocol is used then there is confidentiality of the downloaded/uploaded configuration files and system images. Auditing/Logging - All access and activity relating to downloading/uploading system images and configuration files are logged via AAA services and filter exception rules. DoS Mitigation - A combination of filtering and CRC-check/MD5-based integrity checks are used to mitigate the risks of DoS attacks. If the software updates and configuration changes are performed via an OOB management system, this is also added protection.

Where the MD5 algorithm is not used to perform data-integrity checking of software images and configuration files, ISPs have expressed an interest in having this functionality. IPsec is considered too cumbersome and operationally difficult to use for data integrity and confidentiality.

Although logging is part of all the previous sections, it is important enough to be covered as a separate item. The main issues revolve around what gets logged, how long are logs kept, and what mechanisms are used to secure the logged information while it is in transit and while it is stored.

Attacks on the logged data can be both from passive or active sources. Passive attacks are possible if someone has the capability to intercept data between the recipient logging server and the device from which the logged data originated. This can be accomplished if a single infrastructure device is somehow compromised and can act as a network sniffer, or if it is possible to insert a new device that acts as a network sniffer. Active attacks are possible for both on-path and off-path scenarios. For on-path active attacks, the situation is the same as for a passive attack, where either a device has to already be compromised, or a device can be inserted into the path. For off-path active attacks, the attacks are generally limited to message insertion or modification that can alter the logged data to keep any compromise from being detected, or to destroy any evidence that could be used for criminal prosecution.

Confidentiality violations can occur when a miscreant intercepts any of the logging data that is in transit on the network. This could lead to privacy violations if some of the logged data has not been sanitized to disallow any data that could be a violation of privacy to be included in the logged data.

For a replay attack to be successful, the logging data would need to first be captured either on-path or diverted to an attacker and later replayed to the recipient.

Logging data could be injected, deleted, or modified by someone in control of intermediate hosts. Logging data can also be injected by forging packets from either legitimate or illegitimate IP addresses.

A man-in-the-middle attack attacks the identity of a communicating peer rather than the data stream itself. The attacker intercepts traffic that is sent between the infrastructure device and the logging server or traffic sent between the logging server and the database that is used to archive the logged data. Any unauthorized access to logging information could lead to the knowledge of private and proprietary network topology information, which could be used to compromise portions of the network. An additional concern is having access to logging information, which could be deleted or modified so as to cover any traces of a security breach.

When it comes to filtering, logging is mostly performed on an exception auditing basis (i.e., traffic that is NOT allowed is logged). This is to assure that the logging servers are not overwhelmed with data, which would render most logs unusable. Typically the data logged will contain the source and destination IP addresses and layer 4 port numbers as well as a timestamp. The syslog protocol is used to transfer the logged data between the infrastructure device to the syslog server. Many ISPs use the OOB management network to transfer syslog data since there is virtually no security performed between the syslog server and the device. All ISPs have multiple syslog servers - some ISPs choose to use separate syslog servers for varying infrastructure devices (i.e., one syslog server for backbone routers, one syslog server for customer edge routers, etc.) The timestamp is derived from NTP, which is generally configured as a flat hierarchy at stratum1 and stratum2 to have less configuration and less maintenance. Consistency of configuration and redundancy is the primary goal. Each router is configured with several stratum1 server sources, which are chosen to ensure that proper NTP time is available, even in the event of varying network outages. In addition to logging filtering exceptions, the following is typically logged: routing protocol state changes, all device access (regardless of authentication success or failure), all commands issued to a device, all configuration changes, and all router events (boot-up/flaps). The main function of any of these log messages is to see what the device is doing as well as to try and ascertain what certain malicious attackers are trying to do. Since syslog is an unreliable protocol, when routers boot or lose adjacencies, not all messages will get delivered to the remote syslog server. Some vendors may implement syslog buffering (e.g., buffer the messages until you have a route to the syslog destination), but this is not standard. Therefore, operators often have to look at local syslog information on a device (which typically has very little memory allocated to it) to make up for the fact that the server-based syslog files can be incomplete. Some ISPs also put in passive devices to see routing updates and withdrawals and do not rely solely on the device for log files. This provides a backup mechanism to see what is going on in the network in the event that a device may 'forget' to do syslog if the CPU is busy. The logs from the various syslog server devices are generally transferred into databases at a set interval that can be anywhere from every 10 minutes to every hour. One ISP uses Rsync to push the data into a database, and then the information is sorted manually by someone SSH'ing to that database.

User Authentication - Not applicable. User Authorization - Not applicable. Data Origin Authentication - Not implemented. Access Control - Filtering on logging host and server IP address to ensure that syslog information only goes to specific syslog hosts. Data Integrity - Not implemented. Data Confidentiality - Not implemented. Auditing/Logging - This entire section deals with logging. DoS Mitigation - An OOB management system is used and sometimes different syslog servers are used for logging information from varying equipment. Exception logging tries to keep information to a minimum.

There is no security with syslog and ISPs are fully cognizant of this. IPsec is considered too operationally expensive and cumbersome to deploy. Syslog-ng and stunnel are being looked at for providing better authenticated and integrity-protected solutions. Mechanisms to prevent unauthorized personnel from tampering with logs is constrained to auditing who has access to the logging servers and files. ISPs expressed requirements for more than just UDP syslog. Additionally, they would like more granular and flexible facilities and priorities, i.e., specific logs to specific servers. Also, a common format for reporting standard events so that modifying parsers after each upgrade of a vendor device or software is not necessary.

Although filtering has been covered under many of the previous sections, this section will provide some more insights to the filtering considerations that are currently being taken into account. Filtering is now being categorized into three specific areas: data plane, management plane, and routing control plane.

Data plane filters control the traffic that traverses through a device and affects transit traffic. Most ISPs deploy these kinds of filters at customer facing edge devices to mitigate spoofing attacks using BCP38 and BCP84 guidelines.

Management filters control the traffic to and from a device. All of the protocols that are used for device management fall under this category and include: SSH, Telnet, SNMP, NTP, HTTP, DNS, TFTP, FTP, SCP, and Syslog. This type of traffic is often filtered per interface and is based on any combination of protocol, source and destination IP address, and source and destination port number. Some devices support functionality to apply management filters to the device rather than to the specific interfaces (e.g., receive ACL or loopback interface ACL), which is gaining wider acceptance. Note that logging the filtering rules can today place a burden on many systems and more granularity is often required to more specifically log the required exceptions. Any services that are not specifically used are turned off. IPv6 networks require the use of specific ICMP messages for proper protocol operation. Therefore, ICMP cannot be completely filtered to and from a device. Instead, granular ICMPv6 filtering is always deployed to allow for specific ICMPv6 types to be sourced or destined to a network device. A good guideline for IPv6 filtering is in the Recommendations for Filtering ICMPv6 Messages in Firewalls .

Routing filters are used to control the flow of routing information. In IPv6 networks, some providers are liberal in accepting /48s due to the still unresolved multihoming issues, while others filter at allocation boundaries, which are typically at /32. Any announcement received that is longer than a /48 for IPv6 routing and a /24 for IPv4 routing is filtered out of eBGP. Note that this is for non-customer traffic. Most ISPs will accept any agreed upon prefix length from its customer(s).

Denial-of-Service attacks are an ever-increasing problem and require vast amounts of resources to combat effectively. Some large ISPs do not concern themselves with attack streams that are less than 1G in bandwidth - this is on the larger pipes where 1G is essentially less than 5% of an offered load. This is largely due to the large amounts of DoS traffic, which continually requires investigation and mitigation. At last count, the number of hosts making up large distributed DoS botnets exceeded 1 million hosts. New techniques are continually evolving to automate the process of detecting DoS sources and mitigating any adverse effects as quickly as possible. At this time, ISPs are using a variety of mitigation techniques including: sinkhole routing, black hole triggered routing, uRPF, rate limiting, and specific control plane traffic enhancements. Each of these techniques will be detailed below.

Sinkhole routing refers to injecting a more specific route for any known attack traffic, which will ensure that the malicious traffic is redirected to a valid device or specific system where it can be analyzed.

Black hole triggered routing (also referred to as Remote Triggered Black Hole Filtering) is a technique where the BGP routing protocol is used to propagate routes which in turn redirects attack traffic to the null interface where it is effectively dropped. This technique is often used in large routing infrastructures since BGP can propagate the information in a fast, effective manner, as opposed to using any packet-based filtering techniques on hundreds or thousands of routers (refer to the following NANOG presentation for a more complete description http://www.nanog.org/mtg-0402/pdf/morrow.pdf). Note that this black-holing technique may actually fulfill the goal of the attacker if the goal was to instigate black-holing traffic that appeared to come from a certain site. On the other hand, this black hole technique can decrease the collateral damage caused by an overly large attack aimed at something other than critical services.

Unicast Reverse Path Forwarding (uRPF) is a mechanism for validating whether or not an incoming packet has a legitimate source address. It has two modes: strict mode and loose mode. In strict mode, uRPF checks whether the incoming packet has a source address that matches a prefix in the routing table, and whether the interface expects to receive a packet with this source address prefix. If the incoming packet fails the unicast RPF check, the packet is not accepted on the incoming interface. Loose mode uRPF is not as specific and the incoming packet is accepted if there is any route in the routing table for the source address. While BCP84 and a study on uRPF experiences detail how asymmetry, i.e., multiple routes to the source of a packet, does not preclude applying feasible paths strict uRPF, it is generally not used on interfaces that are likely to have routing asymmetry. Usually for the larger ISPs, uRPF is placed at the customer edge of a network.

Rate limiting refers to allocating a specific amount of bandwidth or packets per second to specific traffic types. This technique is widely used to mitigate well-known protocol attacks such as the TCP-SYN attack, where a large number of resources get allocated for spoofed TCP traffic. Although this technique does not stop an attack, it can sometimes lessen the damage and impact on a specific service. However, it can also make the impact of a DoS attack much worse if the rate limiting is impacting (i.e., discarding) more legitimate traffic.

Some ISPs are starting to use capabilities that are available from some vendors to simplify the filtering and rate limiting of control traffic. Control traffic here refers to the routing control plane and management plane traffic that requires CPU cycles. A DoS attack against any control plane traffic can therefore be much more damaging to a critical device than other types of traffic. No consistent deployment of this capability was found at the time of this writing.