Introduction
This article presents:
- An AIX kernel extension to permit a specified percentage of TCP/IP packets to and from a designated host to be dropped randomly so as to simulate adverse network conditions.
- A utility to load, activate, and unload the kernel extension.
- C and Java™ utilities to monitor total packet throughput to the target host and actual packets dropped.
- Full source for the kernel extension with C and Java utilities so that the software can be built, customized, and developed according to local requirements.
All the components needed to build and run this packet dropping
simulator are provided with this article as a downloadable compressed
file. The How to build and use the kernel extension as supplied section enables you to get started and use it as soon as possible.
The kernel extension is designed to be activated on a single
source node from which it will selectively drop packets either outbound
to a target node or received from it. It is not necessary to install the
packet drop simulator on the target node. The target node itself does
not even need to be AIX-based but just needs to support the TCP/IP
protocol.
The article first presents some background to TCP/IP performance
and why dropped packets can have a serious impact on application
performance. You will then see how to build and use the kernel extension
as provided. This enables you to get up and running quickly with it.
The supplied functionality is deliberately very simple. It allows
you to set a given percentage of TCP/IP packets to be dropped randomly
to and from a specified host, with all other hosts and protocols
unaffected and no retained history. Because in practice, you might need
to add further functionality into the kernel extension according to your
own requirements, you will also be shown details as to how the
extension works and how it can be customised. You can then decide to do
this if you need to:
- Permit packet dropping to and from multiple hosts.
- Support different packet drop rates for each of those hosts.
- Create historic logs of total packets received, transmitted, and dropped in both directions.
- Implement the packet drop simulator on different versions of AIX or C compilers to those used when the software was written.
The kernel extension could be readily developed further to
simulate other network issues that could give rise to performance
problems, such as packet corruption, packets arriving out of order, and
jitter.
It is recommended the kernel extension is only used on non-production systems.
Background
First let's review some basic terms that are important when considering network performance, bandwidth, latency, and congestion.
Bandwidth is the capacity of the network, that is
the rate at which data can be pushed across it. This is usually measured
in Megabits per second (Mbps), where one Megabit is 10^6 bits. It can
be thought of as the width of the network pipeline.
Latency is the time it takes for one piece of data
to traverse the network. This is usually measured in milliseconds. It
can be thought of as the length of the network pipeline.
The difference between bandwidth and latency is illustrated in Figure 1.
Figure 1. Latency and bandwidth and the network pipeline
Congestion is the effect of heavy usage of the
network. It can result in delays in propagating data, loss of packets
resulting in packet retransmission, and an inability to make connections
to the network. Packet loss is usually measured as a percentage. TCP
can normally handle losses up to 0.1% with little direct impact. If the
losses increase above this, the effect can be more severe.
Network latency, bandwidth, and congestion can have a serious
impact on application performance. Consider, for example, the situation
where a front-end application on one system makes frequent calls across
the network to another system which hosts a database used by the
application. The traffic across the network to the database might
consist of many bursts of small data, or might consist of fewer sets of
relatively large data transfers. Either way, the ability of the network
to transfer this data reliably and speedily can be a significant factor
in the application's perceived end-to-end performance by a customer.
Application development teams often have access to isolated,
high-speed networks that can be much more lightly used than the
environment into which an application is to be deployed. This can mean
the effect of poor network performance on the application may never be
observed within the environment in which it is developed and tested.
This can, in turn, mean that the performance observed within a
customer's environment might be very different to that observed in the
environment from which it is tested.
This article focuses on the effect of dropped packets on a
particular application's performance. Due to the fact that there can be
very high numbers of packets transferred between a front-end and a
back-end system and the patterns in which these application packets are
transmitted are very specific to the nature of the front-end and
back-end applications, the simplest approach to understand the impact on
the application is to simulate the dropping of packets.
Packets can be dropped when transferring data between systems for two key reasons:
- Heavy network utilization and resulting congestion
- Faulty network hardware or connectors
TCP is designed to be able to react when packets are dropped on a
network. When a packet is successfully delivered to its destination, the
destination system sends an acknowledgement message back to the source
system. If this acknowledgement is not received within a certain
interval, that may be either because the destination system never
received the packet or because the packet containing the acknowledgement
was itself lost. In either case, if the acknowledgement is not received
by the source system in the given time, the source system assumes that
the destination system never received the message and retransmits it. It
is easy to see that if the performance of the network is poor, packets
are lost in the first place, and the increased load from these
retransmit messages is only increasing the load on the network further,
meaning that more packets will be lost. This behaviour can result in
very quickly creating a critical situation on the network.
There are some freely available sophisticated software products
that can help to simulate different characteristics of varying network
performance. These include:
- WANem: Runs from Knoppix Linux® based CD
- ipfilter: Public domain software but not fully ported to AIX
- dummynet (FreeBSD)
- ALTQ (FreeBSD)
WANem runs from a Knoppix-based CD and can simulate a large range
of different network characteristics. It does offer the ability to drop a
random number of packets with filtering options but would need to run
on a separate system in between the source and target system.This may
not be convenient, for example if the source and target systems are
installed on dedicated high-speed networks.Building the kernel extension
and utilities
Although ipfilter looked very promising, it had only been
partially ported to AIX 5.3 and it became evident that it would be
quicker to write a bespoke utility providing the minimal functionality
required rather than porting the whole of ipfilter to both AIX 6.1 and
7.1.
dummynet is a component of the FreeBSD operating system and has
also been ported to other platforms including Linux and Microsoft®
Windows®. The disadvantage of using this though is that to simulate
packet drops for packets originating from an AIX host, an intermediate
FreeBSD/Linux host has to be configured and set up with dummynet.
ALTQ is an alternate queueing framework that helps to provide bandwidth control, mostly on BSD routers.
There are also other packages commercially available such as LANforge ICE but these were ruled out for cost reasons. LANforge also requires dedicated hardware.
How to build and use the kernel extension as supplied
Requirements
The packet drop simulator was built and tested in the following environments:
- AIX V7.1.4.0 with gcc V4.7.2-1
- AIX V6.1.2.0 with gcc V4.2.0
- AIX V6.1.4.0 with gcc V4.2.0
The kernel extension and associated C programs were compiled using
the gcc compiler. It should be straightforward to use an AIX C compiler
instead if required, although some of the compilation flags will need
to be changed.
Note that it will be necessary to install the AIX bos.adt.syscalls file
set to build the kernel extension. This will be available on your AIX
installation media. The file set will not be required on systems were
the kernel extension is only to be deployed as long as the AIX level is
identical to the build system. If there are any differences in AIX
levels between the two systems, the kernel extension should be built on
each.
The kernel extension, the kctrl utility to load and activate it and the C example control and monitor are all built with the AIX makefacility. A makefile to build all of these components is included in the downloadable zip file.
Precautions
Two important factors should be kept in mind before using this utility:
1) Care should be exercised when developing and testing any kernel
extension. It is common for errors to result in system failure. A
dedicated test system is ideal, but avoid using systems that are also
used for other purposes or by other users. If the system is located in a
remote data centre, you should also ensure you have the means for it to
be rebooted when required if it does not reboot automatically.
2) Avoid the use of flood pings to test the utility on a shared network as this may severely impact other users.
Building the kernel extension and utilities
First make sure you have met the requirements on your development system as outlined above.
Next, extract the downloadable attachment into a working directory of your choice on the target AIX system.
The download does not include pre-built binaries so as to avoid
the risk of running it on incompatible operating system levels. The
software will therefore need to be built once it has been extracted. To
this, change directory to the working directory you created above and
then run the make commands as follows:
# make
gcc -Wall -maix64 -ffreestanding -msoft-float -o pdrop_kernex64.o -c
pdrop_kernex.c
ld -b64 -o pdrop_kernex64 pdrop_kernex64.o -e pdrop_init -bI:
/home/jerry/kernext/kernex.exp -bI:/home
/jerry/kernext/netinet.exp -bE:/home/jerry/kernext/pdrop_syscall.exp -lsys -lcsys
rm -f pdrop_kernex
ar -X64 -r -v pdrop_kernex pdrop_kernex64
ar: Creating an archive file pdrop_kernex.
a - pdrop_kernex64
gcc -Wall -maix32 -o pdrop_ccm32 .c -Xlinker -bI:
/home/jerry/kernext/pdrop_syscall.exp
gcc -Wall -maix64 -o pdrop_ccm64 pdrop_ccm.c -Xlinker -bI:
/home/jerry/kernext/pdrop_syscall.exp
gcc -maix64 -o kctrl kctrl.c
gcc -Wall -I/usr/java6/include -I. -shared -fPIC pdrop_jni.c -o libpdrop_jni.so
-Xlinker -bI:/home/jerry/kernext/pdrop_syscall.exp
Target "all" is up to date.
#
After the make command is run successfully, all C components of
the packet drop utility required to use it will be ready for use.
Note that the Java control and monitor application need to be built separately as described in the Control and monitor applications section
Loading and activating the kernel extension
One of the binaries that will have been build in the proceeding step is the kctrl program.
This permits the kernel extension to be loaded, activated, and
unloaded. To load and activate the kernel extension, proceed as follows:
# ./kctrl /home/jerry/kernext/pdrop_kernex64 Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd l Extension Successfully loaded, kmid is 1353523200 Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd i Extension Initialized Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd e #
At this point, the kernel extension will be loaded and active, but
will not drop any packets to a target host until either the C or Java
control and monitor utilities are used.
Initiating packet dropping and monitoring
Having loaded and initialized the kernel extension, you can now see how to use the supplied C utility, pdrop_ccm, to control and monitor the setting of packets to a target host.
pdrop_ccm is invoked with arguments that specify the
location where the kernel extension was built, the target host name, and
the drop rate.
# ./pdrop_ccm64 /home/jerry/kernext/pdrop_kernex64 Usage: ./pdrop_ccm64 <kernel_extension> <TargetIP> <drop_percentage> #
You can use this, for example, with the host name fred and a drop rate of 0.5% as follows:
# ./pdrop_ccm64 /home/jerry/kernext/pdrop_kernex64 fred 0.5 Kernel extenstion is loaded. Setting drop ip to: x.y.z.l Drop rate: 0.50. Total In In dropped PC in dropped Total Out Out dropped PC out dropped 0 0 0.00 0 0 0.00 10120 57 0.56 10174 53 0.52 22224 134 0.60 22340 116 0.52 #
In this example, shortly after we started pdrop_ccm we started sending IP traffic to the target host. The pdrop_ccm utility will keep running until it is interrupted with a ctrl+c and will print an update line every 10 seconds.
Please note that the first command line parameter, the kernel
extension name, must be specified exactly as it was when loaded with
kctrl.
The drop percentage argument is optional. If it is not supplied,
pdrop_ccm will display the current drop rate only and will not re-set
it.
A Java-based control and monitor utility is also provided and this is described in the Control and monitor applications section.
That is all there is to get started with the kernel extension.
The rest of this article presents more details on how the kernel
extension works and how it and the control and monitor applications can
be developed and customized for individual use.
How the kernel extension works
The AIX kernel extension is written in the C programming language.
The kernel function exposes access to it through a set of system calls.
The C and Java control and monitor applications utilize these system
calls to drive the kernel extension to drop packets and to and from a
particular host at a given rate. For the Java environment, a JNI wrapper
class provides access to the kernel extension's system calls from Java.
The kernel extension itself consists of:
- A set of system calls which control the dropping of packets and the collection of packet statistics
- Two functions which hook into the kernel's networking layers. One of these is for inbound traffic and the other for outbound. These functions are called whenever a packet is received or is to be dispatched to control the dropping of packets.
The system calls are accessed by the user-level b to set the
remote system to which incoming and outgoing packets should be dropped
and what percent of packets should be dropped. These system calls also
contain functions to retrieve counters defining how many packets have
been dropped in both inbound and outbound directions and also a function
to reset the counters.
For the C environment, these system calls are used directly in the
example control and monitor application pdrop_ccm.c, which we used in
the Initiating packet dropping and monitoring section
above. For the Java environment, the system calls can be used using
JNI. The JNI wrapper can then be used by a bespoke Java application to
control and monitor the dropping of packets. Again, a sample Java
application is provided as a downloaded resource with this article. This
is called pdrop_jcm.java. Refer to the Java control and monitor section for further details on the example Java utility.
Figure 2 illustrates the different components to the kernel
extension and the flow of data through it. The following two sections
discusses the flow of packet flow through the extension, both outbound
data to be sent to the target node and inbound data received from it.
Outbound traffic flow
First, consider the case of outbound traffic, where an application
under test submits data to be transmitted across the network. Refer to
point A in the Figure 2. The application submits a set of data to the
kernel for sending to the remote node on the network. The TCP subsystem
breaks this down into a number of packets which are normally then sent
to the appropriate network device driver for dispatch across the
network. When using the kernel extension, however, the hook outbound_fw
is set in the IP layer which results in the kernel extension's outbound
filter function pdrop_outbound_filter being called for
every packet being dispatched. This retrieves the IP header from the
packet and matches it against the target node for which packets are to
be dropped. If the node does not match, the function, ip_output_post_fw is
called which routes the packet to the appropriate interface on the
host. If the node does match, then a random decision whether to pass the
packet is made, which is weighted according to the number of packets to
be dropped. If this decision is that the packet should be sent, the ip_output_post_fwfunction
is called and the packet is passed. Otherwise, the mbuf holding the
data is freed and the outbound filter returns and the packet is not
sent.
Inbound traffic flow
For inbound traffic, a corresponding operation applies when data
is received from the network by the device driver. Refer to point B in
Figure 2. The device driver forwards the packet of data to the TCP/IP
subsystem. This now calls the inbound filter which again has been set by
the kernel extension by assigning the inbound hook inbound_fw to the pdrop_inbound_filterfunction.
This function makes the decision whether to pass the packet or not. If
the packet is not from the target node for which packets are being
dropped, it is automatically passed by calling the ipintr_noqueue_post_fw function.
If it is from the target node, then again a random decision is made
whether to pass the packet according to the required percentage of
packets that are to be dropped. If the decision is to send the packet,
the ipintr_noqueue_post_fw function is called and
the packet is passed. Otherwise, the mbuf holding the data is freed and
the inbound filter returns and the packet is not passed up to the higher
layers.
The AIX IP filtering hook and kernel services described above are explained in the IBM AIX 7.1 information center.
Figure 2. The packet drop kernel extension
Inbound filter
This is the inbound filter function:
/*
** This function is the filter for incoming traffic
*/
void pdrop_inbound_filter(ifp, m, args)
struct ifnet *ifp;
struct mbuf *m;
inbound_fw_args_t *args;
{
struct ip *ip_in;
if (PDROP_TRUE == amDropping)
{
ip_in = mtod(m, struct ip *);
if ( (ip_in->ip_p == IPPROTO_TCP) &&
(dropip.s_addr == ip_in->ip_src.s_addr))
{
fetch_and_addlp((atomic_l) &total_in, 1);
if (dropMod != 0)
{
if ( (pdrop_random() % dropMod) == 1)
{
fetch_and_addlp((atomic_l) &nin_dropped, 1);
if (m != NULL)
{
m_freem(m);
}
return;
}
}
}
}
ipintr_noqueue_post_fw(ifp, m, args);
return;
}
The arguments to the filter function include an mbuf containing
the received data. The mtod macro is used to convert the pointer to the
mbuf into a pointer to the received data, which is addressed as an IP
header. A counter of the total number of received packets from the
target address is maintained.
The function then compares the IP address selected for dropping to
the source address specified in the received packet. If these match,
then the pdrop_random() function is called, which uses the modulus
function to select the required percentage of packets for dropping. If
the packet is to be dropped, then an internal counter of dropped packets
is incremented. If it is not to be dropped, then the ipint_noqueue_post_fw function is called to pass the packet to TCP for processing.
The fetch_and_addlp system functions are called to
increment the counts of the total number of received and dropped
packets. These ensure that the counts are maintained atomically so as to
be thread safe.
You can see from this example that it is possible to specify which
protocol is to filtered. In this case, we are only dropping packets for
the TCP protocol. For testing, it is a good idea to make it use another
protocol, such as Internet Control Message Protocol (ICMP) so then the
packet dropper can be used with commands, such as ping. To do that, the IPPROTO_TCP constant in the test above would be changed to IPPROTO_ICMP.
Note: amDropping, dropip, and dropMod in the above extract are defined as global variables.
Outbound filter
This is the outbound filter function:
/*
** This function is the filter for outbound network traffic
*/
int pdrop_outbound_filter(ifp, m, args)
struct ifnet *ifp;
struct mbuf *m;
outbound_fw_args_t *args;
{
struct ip *ip_out;
if (PDROP_TRUE==amDropping)
{
ip_out = mtod(m, struct ip *);
if ( (ip_out->ip_p == IPPROTO_TCP) &&
(dropip.s_addr == ip_out->ip_dst.s_addr))
{
fetch_and_addlp((atomic_l) &total_out, 1);
if (dropMod != 0)
{
if ( (pdrop_random() % dropMod ) == 1)
{
fetch_and_addlp((atomic_l) &nout_dropped, 1);
if (m != NULL)
{
m_freem(m);
}
return 0;
}
}
}
}
ip_output_post_fw(ifp, m, args);
return 0;
}
This works very similar to the inbound filter. Counters are
maintained for the total number of packets destined for the target IP
address and for the total number of outbound packets dropped.
The pointer to the mbuf is freed if the packet is to be dropped.
Note: amDropping, dropip, and dropMod in the above extract are defined as global variables.
Randomizing the packets to be dropped
It is important for the kernel extension to select packets at
random for dropping. If say, packets were dropped after a set number of
packets had been sent, this would not necessarily be a true emulation of
what happens on a heavily used network. Further, it is possible that
the software under test may behave differently when it misses packets at
regular and irregular intervals.
The standard C library rand() call to generate random
numbers cannot be used in a kernel extension. This is because such
functions are not safe to use in a the re-entrant kernel environment. If
you attempt to use this function, the system might fail when it is
called from the kernel extension.
A simple function to mimic a call to random was therefore used.
This generates a long random number based on the following inputs:
- The seconds field from the current time
- The nano-seconds field from the current time
- The number of calls to the random function
- The number of processor ticks since system boot
This is the source of the random function that is used in the kernel extension.
long pdrop_random()
{
static long calls = 0;
struct timestruc_t ts;
long x = 0;
curtime(&ts);
// lbolt is the number of clock ticks since boot, see HZ in /usr/include/sys
/m_param.h for number of ticks/sec
fetch_and_addlp((atomic_l) &x,
(long) ts.tv_nsec + (long) ts.tv_sec +
(long) lbolt +
calls++);
return x;
}
This is not of course a true random function but is a good enough
mechanism for assuring that packets are dropped in a sufficiently
irregular manner. As the calls variable is static, it is incremented atomically with the fetch_and_addlp kernel service.
System calls provided in the packet drop kernel extension
In this section, you are shown details of the various system calls
provided in the kernel extension. These will be of interest if you need
to write your own control and monitor applications or customize the
kernel extension.
Setting the target address: int pdrop_set_drop_address(struct in_addr *block_addr)
This function takes a pointer to an in_addr struct as an argument. The method may be called from the application level from a string-based host name as follows:
int setDropAddress(char *hostname)
{
int status = 0;
struct in_addr block_addr;
if (1 == inet_aton(hostname, &block_addr))
{
if (PDROP_TRUE == pdrop_set_drop_address(&block_addr))
status = 1;
}
return status;
}
Retrieving the target address: int pdrop_get_drop_address()
This function returns the target address as a 32-bit unsigned integer.
Setting the rate at which packets are dropped: extern void pdrop_setDropMod(long m);
This function takes a long input value, which represents how often
packets should be dropped. The way this is worked out is that the
kernel extension generates a random number and then takes a modulus of
that random number with the long value. If the result is one, then the
packet is dropped, otherwise it is passed.
The call is designed in this way as it is not possible to perform floating point operations within the kernel extension.
Here is a simple way of callingpdop_setDropMod()from the application level based on a double percentage value:
void setDropPC(double d)
{
long x = 0;
x = 100.0 /d;
pdrop_setDropMod(x);
}
So, for example, if it was required to drop 0.1% of packets, then d would be 0.1 and the long value passed topdrop_setDropModwould be 1000.
Retrieving the rate at which packets are dropped:extern long pdrop_getDropMod();
This function returns the long representation of how often packets
should be dropped, which is expressed in the same way as a call topdrop_setDropMod();
Activate the dropping of packets: void pdrop_startDropping()
When the kernel extension is started, packets are not dropped
until the pdrop_startDropping() function is called. You should call this
function after setting the target address and the rate at which packets
should be dropped.
Stop the dropping of packets: void pdrop_stopDropping()
This function stops the dropping of packets. The functionspdrop_startDropping()andpdrop_stopDropping()can be called as required to enable and disable packet loss.
Query whether packets are being dropped:int pdrop_amDropping()
This function returns a 1 if the kernel extension is currently dropping packets and a 0 it is not.
Retrieving kernel extension counters:void pdrop_getstats(struct pdrop_stats *p)
This function returns a structure which contains details of:
- The total number of packets received from the target address
- The total number of packets received from the target address which were dropped
- The total number of packets sent to the target address
- The total number of packets sent to the target address which were dropped.
The definition of the structure used is in the header file, kernext_pdrop.h.
Here is an example of how the counters might be received from the application level:
pdrop_stats_t j;
pdrop_getstats(&j);
printf("Total in: %d.\n", j.total_in);
printf("In dropped: %d.\n", j.nin_dropped;);
printf("Total out: %d.\n", j.total_out);
printf("Out dropped: %d.\n", j.nout_dropped);
Resetting the internal kernel extension counters:void pdrop_reset_counters()
This function is used to reset the counters returned bypdrop_getstatsto zero.
Configuring the kernel extension
TheHow to build and use the kernel extension as supplied section provided information about how to build the kernel extension and its utilities. This section provides:
- More details on the build process
- Information on AIX 6.1 and AIX 7.1 build issues
- Details about exporting the system calls provided in the kernel extension to the application level
- Details about loading and activating the kernel extension
- Information about the use of system logging to record when the extension is loaded and unloaded
AIX 6.1 and AIX 7.1 build issues
Beginning with AIX 6.1, the AIX operating system simplified its
kernel environment by providing only the 64-bit kernel. AIX 6.1 and AIX
7.1 maintain application binary compatibility with previous AIX versions
as specified above, but device drivers and kernel extensions that are
only 32-bit cannot be built on AIX 6.1 or AIX 7.1.
As this article presents the kernel extension as suitable for AIX 6.1 and AIX 7.1, it has been built in the 64-bit mode.
The build process
A Makefile is included with the compressed file provided with this article. This can be used to build the kernel extension and application layer programs associated with it. Note that this Makefile has been written for use with the AIX make utility and it will need to be modified if you wish to use gnu make.
The kernel extension itself is built with the following commands:
gcc -maix64 -ffreestanding -msoft-float -o pdrop_kernex64.o -c pdrop_kernex.c ld -b64 -o pdrop_kernex64 pdrop_kernex64.o -e pdrop_init -bI:/usr/lib/kernex.exp -bI: /usr/lib/netinet.exp -bE:/home/jerry/kernext/pdrop_syscall.exp -lsys -lcsys
As noted earlier, the kernel extension is built as a 64-bit binary. The -ffreestanding and -msoft-float options
are used to prevent the use of floating point instructions to
manipulate certain data structures. This was required on AIX 7.1, but
was not necessary on AIX 6.1.
The ld command specifies the entry point into the kernel
extension, which in this case is the pdrop_init() function. This
function is called when the kernel extension is activated.
The ld command also refers to the files kernex.exp and netinet.exp. These files are provided with the bos.adt.syscalls file
set. This file set may not be installed on your test system in which
case you will need to ask the system administrator to install it.
Note that it is necessary to make one small modification to the /usr/include/sys/socketvar.h header
file for gcc to compile the kernel extension satisfactorily. You should
retain a safe copy of the file and locate the following line:
extern struct free_sock_hash_bucket free_sock_hash_table[];
This should be changed to:
extern struct free_sock_hash_bucket * free_sock_hash_table;
The kernel extension makes this #define in the source:
#define _MSGQSUPPORT 1
This is necessary to prevent the fd_select() system call being made which is not available to the kernel extension. Using#define changes the fd_select() call to the original select() call, which is available to the kernel extension.
Exporting the provided system calls
The pdrop_syscall.exp file defines the kernel extension system
calls that are exported to the application level. The contents of this
file are as follows:
#!/unix pdrop_set_drop_address syscall3264 pdrop_get_drop_address syscall3264 pdrop_setDropMod syscall3264 pdrop_getDropMod syscall3264 pdrop_startDropping syscall3264 pdrop_stopDropping syscall3264 pdrop_amDropping syscall3264 pdrop_getstats syscall3264 pdrop_reset_counters syscall3264
If you need to customize the kernel extension to include further system calls, you will need to amend this file. The syscall3264identifier
at the end of each line makes the system calls available to both 32-bit
and 64-bit processes. The flag may also be set to syscall32 to support calls from 32-bit processes only or syscall64 for
64-bit processes only. If the flag is not set for the correct target
process environment, the process will fail with a segmentation fault
when the system call is made. For further details on how to set this
identifier refer to the topic, Exporting Kernel Services and System Calls.
Use of kctrl to control loading of the kernel extension
The kernel extension is loaded and unloaded with the kctrl program
provided as described earlier. The program should be invoked with the
full path name of the kernel extension. It then interactively accepts
the following commands:
- q – checks whether the kernel extension has been loaded
- l – loads the kernel extenstion
- I – initializes the kernel extension
- t – terminates the kernel extension
- u – unloads the kernel exension
- e – exits the utility
Here is an example where kctrl is used to query, load, initialize, terminate, unload, and quit the utility:
# ./kctrl /home/jerry/kernext/pdrop_kernex Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd q Extension is not loaded Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd l Extension Successfully loaded, kmid is 1353052160 Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd i Extension Initialized Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd q Extension is loaded, with kmid 1353052160 Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nd e #
You can also check whether the kernel extension has been loaded by using the AIX genkex command, for example:
# genkex | grep -i kern f1000000c02be000 2000 /home/jerry/kernext/kernext_hello #
In this example, the kctrl executable works with interactive user input. It would be straightforward to modify kctrl to
take the command as another argument on the command line and thereby
make it easy to incorporate within system startup and shutdown scripts.
However, this is not generally recommended given the nature of the
utility.
Logging in the kernel extension
The kernel extension uses the syslogd daemon to record when
the extension is loaded and unloaded. This can be useful for debugging
or auditing purposes. To enable this logging:
- Ensure that logging is enabled in /etc/syslog.conf. For example, the following line can be appended to this file:
- Ensure that the log file enabled in /etc/syslog.conf exists.
- Refresh the syslogd subsystem (refresh -s syslogd)
Use with other IP protocols
Although the extension is primarily considered for TCP/IP, it
would be simple to change it to work with other IP-based network
protocols. This was discussed in the How the kernel extension works section.
Control and monitor applications
Two example applications that can use the system calls exposed by
the kernel extension are provided. One is for the C environment and the
other for the Java environment.
These applications show you how to write your own custom
applications to control packet dropping. You can decide to do this if
you need to write your own automated test framework to simulate
different network conditions.
It should be noted that only the applications that collect
statistics directly from the kernel extension can provide meaningful
statistics on packets that have been dropped. Operating system provided
utilities that report on dropped packets will not include details of
packets that have been dropped through the kernel extension. This is
because the packets are dropped by the extension before they are passed
through the TCP/IP subsystem for dispatch or delivery.
C control and monitor
The test application, pdrop_ccm.c, is provided in the downloads
section of this article. This is invoked with a target host name and
drop percentage on the command line. The application passes details of
the target host and the drop rate to the kernel extension and then
monitors the inbound and outbound packets to the target system every 10
seconds.
The packet drop system calls can be invoked from both 32-bit and
64-bit applications. The makefile provided demonstrates this by building
both 32-bit and 64-bit versions of the control and monitor application.
These are called pdrop_ccm32 and pdrop_ccm64 respectively.
pdrop_ccm shows the total number of input and output
packets, the number of input and output packets that were dropped, and
the numbers of dropped packets as a percentage. It takes the name of the
kernel extension, the name of the target host as arguments and an
optional value specifying the target drop rate. If this third parameter
is not specified on the command line, no change to the drop rate is
made. Here is an example of the application in use:
# ./pdrop_ccm /home/jerry/kernext/pdrop_kernex64 fred 1.0
Extension is loaded, with kmid 1353052160
Official name is: fred
IP addresses: 9.20.XXX.YYY
Target IP address: 9.20.XXX.YYY.
Total In In dropped PC in dropped Total Out Out dropped PC out dropped
0 0 0.00 0 0 0.00
0 0 0.00 0 0 0.00
13138 138 1.05 13283 144 1.08
29158 275 0.94 29460 302 1.03
43738 423 0.97 44206 467 1.06
58320 564 0.97 58952 632 1.07
72653 721 0.99 73437 784 1.07
87227 872 1.00 88153 926 1.05
#
Here, we can see that the host to which packets are to be dropped
is fred and 1.0% of the packets are to be dropped. Status updates on
total and dropped packets are then displayed at 10 second intervals.
The test application calls the pdrop_reset_counters() method
to reset the kernel extension counters. After 10 seconds, you can see
that network activity to the target system is started and the packet
statistics are displayed.
Note that the example controls a single target system only. If you
run the pdrop_ccm command again and specify a different target host
name, packets to and from the original host will no longer be dropped.
Java control and monitor
A sample Java application, PDrop_jcm.java, is also provided, which demonstrates how to use the packet drop simulator from the Java environment.
PDrop_jcm.java uses JNI to access the C environment for controlling and monitoring the kernel extension. The shared librarylibpdrop_jni.so provides
native methods which can be called from the Java environment. This
shared library is built as part of the build process described earlier
from the source, pdrop_jni.c.
The JNI wrapper provides the following native methods at the Java level which map to the functions in the shared library:
public native boolean isExtensionLoaded(String extName);
public native boolean setDropAddress(String hostname);
private native String getDropAddress();
public native boolean amDropping();
public native int setDropping(boolean doDrop);
public native void resetCounters();
public native void setDropHostName();
public native void setDropPC(float d);
public native float getDropPC();
public native long getTotalIn();
public native long getTotalOut();
public native long getInDropped();
public native long getOutDropped();
When these native methods are invoked, the corresponding function
in the shared library will be called and the result will be returned to
the Java environment.
To build and use this test Java application, first, make sure that
you have set the PATH to the Java SDK environment correctly, for
example:
export PATH=/usr/java6/bin:${PATH}
Next, set the LIBPATH environment variable to reference the directory where the kernel extension is located, so the shared librarylibpdrop_jni.so can be resolved:
export LIBPATH=/home/jerry/kernext:/usr/lib
Now compile the PDrop_jcm.java application, for example:
javac -d . PDrop_jcm.java
Optionally, if you need to regenerate the JNI header file, run the
following command. It mightnot be necessary to run unless you have
customized the kernel extension and changed or added to the native
methods:
javah -d . PDrop_jcm
If you have regenerated the header, you will need to rebuild the shared library using the process described earlier.
The Java application is then run with the path to the kernel
extension, target host name, and optional new drop rate as arguments, as
in the C example above:
# java PDrop_jcm /home/jerry/kernext/pdrop_kernex64 fred 1.0 Ext name: /home/jerry/kernext/pdrop_kernex64. Extension is loaded, with kmid 1353052160 Drop address is: 9.20.XXX.YYY Drop PC: 1.0 Dropping enabled Total In In dropped PC in dropped Total Out Out dropped PC out dropped 0 0 0.0 0 0 0.0 0 0 0.0 0 0 0.0 13189 145 1.09 13307 118 0.88 28515 295 1.03 28782 267 0.92 44306 461 1.04 44701 395 0.88 59216 609 1.02 59753 537 0.89 72979 756 1.03 73688 709 0.96 88606 910 1.02 89455 849 0.94 102550 1054 1.02 103567 1017 0.98 104522 1071 1.02 105551 1029 0.97 104522 1071 1.02 105551 1029 0.97
In this example, you can see that when monitoring started there
were zero packets sent or received to or from the target host, but there
was then network activity that caused packets to be dropped in both
directions. In this example, the drop rate had been configured at 1%.
Measuring non-simulated packet loss
It was mentioned earlier that operating system utilities should
not be used to monitor packets dropped through the use of the kernel
extension. However, these will be required when you need to access
packets that are actually dropped across the network. This section gives
some useful hints and tips for using these utilities.
The ping utility displays the packet loss on the ICMP
packets being sent across the network. You can see the packet loss
statistics in the above example, where it is zero. However, this
measurement is only based on these packets being transferred from and to
the ping utility and do not measure any other packets traversing the network.
On some systems, ping also supports an option to write ICMP
packets as fast as possible onto the network and again gives you the
loss statistics at the end. This is a so called flood ping. You
should use this with caution as it is likely to impact general network
performance while running. It should be run only for a few seconds and
only under test conditions.
The netstat utility can also be used to see how many packets have been dropped on a network. Here is an example output from netstat -D on AIX.
Using netstat to report packet drops (AIX):
# netstat -D
Source Ipkts Opkts Idrops Odrops
-------------------------------------------------------------------------------
ent_dev0 26820670 16079610 0 0
---------------------------------------------------------------
Devices Total 26820670 16079610 0 0
-------------------------------------------------------------------------------
ent_dd0 26820670 16079610 0 0
---------------------------------------------------------------
Drivers Total 26820670 16079610 0 0
-------------------------------------------------------------------------------
ent_dmx0 26820664 N/A 6 N/A
---------------------------------------------------------------
Demuxer Total 26820664 N/A 6 N/A
-------------------------------------------------------------------------------
IP 50335154 44013816 1110616 147099
IPv6 1473 1473 0 0
TCP 43110957 40986657 6235 0
UDP 6105902 2045877 5022268 0
---------------------------------------------------------------
Protocols Total 99552013 87046350 6139119 147099
-------------------------------------------------------------------------------
en_if0 26820664 16079528 0 0
lo_if0 27970975 27975366 4652 0
---------------------------------------------------------------
Net IF Total 54791639 44054894 4652 0
-------------------------------------------------------------------------------
NFS/RPC Client 23 N/A 0 N/A
NFS/RPC Server 0 N/A 0 N/A
NFS Client 14949 N/A 8 N/A
NFS Server 0 N/A 0 N/A
---------------------------------------------------------------
NFS/RPC Total N/A 14977 8 0
-------------------------------------------------------------------------------
(Note: N/A -> Not Applicable)
#
On AIX, the netstat -Zs -p tcp command can be used to reset the protocol statistics before running the promotion activity.
If packet drops are consistently in excess of 0.1%, then you should raise this with your network administrator.
Retransmitted packets can be seen using the following command:
$ netstat -s -p tcp | grep retrans
Statistics of interest from the output of this command are:
- Packets sent
- Data packets
- Data packets retransmitted
- Packets received
- Completely duplicate packets
- Retransmit timeouts
Conclusion
This article provided reference material and instructions to
build, use, and customize a simple utility to simulate dropped TCP
packets on AIX. Such a utility is invaluable when writing cross network
software to model how it will behave under non-ideal network conditions.
The tool may be adapted as required and can easily be enhanced to
support simulation of other network issues that can give rise to
performance problems, such as packet corruption, packets arriving out of
order, and jitter.
How can I obtain the source code mentioned in the article?
ReplyDelete