August 14th 2013

Contact
o o o

Name: Phone:

e-mail:

Due Date: Status

Latest Update: RM Pending: Challenges: Completion Date Change:

Closed Date: Files

\\10.51.46.101\network\ In forl SBC Network Diagram-2007.vsd der SBC upload captures an everything here

SBC Network Diagram-2007.pdf

Instructions:
Bridge: US: 866-692-4541 Mexico: 001-866-597-6485 India: PASSCODE INFORMATION: Leader Passcode: 310-6059 Participant Passcode: 664-4720

1800-300-00468

I will send to F5 and Netscout that the tests and captures start at 11:00am We can start to capture on the f5 & netscout from 11:00am to noon. Get that capture and then send f5 and Netscout. we just need to get captures from the F5 and Neetscout o The load balancer are set to standalone. The load balancer we are using at the moment is eordsc-di0b-lb. This LB has been set as a standalone. It has been upgraded to to 11.2 build 2557.0 hotfix 2. o This is the command that you need to run on the F5 to get the dumps: o tcpdump -ni 0.0:nnn host 10.52.24.68 or 10.51.66.25 or 10.51.66.26 and port 443 -w /var/tmp/1-274858283-9_ssl.pcap

After each capture you get please rename it, by adding a number to the name eg. 1274858283-10_ssl.pcap o To access the lb your need to access as root, pass= Plunder!16 To get the captures from the netscout just select if4 and set the time from 11:00am to noon, then on hte filter you will put source 10.51.66.25 for port 443

we need f5 to let us know why we see retransmission from on of the SNAT pool addresses 10.52.27.1 trhu .25 to the servers These are the Ips that have issues: 10.51.66.25 10.51.66.26 VIP 10.52.24.68 . Don0t let F5 go until they give you a answer your comfortable with.

you need to get the captures an name them acording to the time you started them like SBC-VIP-10.52.24.68-Aug-17-at-9-20am an place them under Coordinate with F5 I will attach here the e-mail here. Just reply when you need help.

Description
Client IP = 172.16.163.108 F5 VIP = 10.52.24.132 eordsc-di0a-lb S/N f5-lgha-ypxh Ticket #: 1-274-858-331 1-274858-283 Edgar opened this one Pool name: sbcprod_https SBC Server IP = 10.51.66.11/23 MAC: 0050.5683.4110 port 1422 GW:
10.51.67.254

SBC Server IP = 10.51.66.12/23 MAC: 0050.5683.4112 port 1422 GW:

10.51.67.254

F5 IP to Backend = 10.52.27.23 The incoming request was logged at 10:11:59am and end around 10:12:15am. Client Side WireShark Capture File 1012.pcapng SBC Backend Server File tcpdump_0539_binary,cap The SSL connection failure terminated by F5 starts at packet #305077 and end with packet #305101. I cannot see this client connection attempt in the TCPDUMP trace file from the SBC server. I do see retransmission between the backend server and the F5 that occurs between 10:11:59am and end around 10:12:15am.

[root@eordsc-di0a-lb:Active] config # traceroute 10.51.66.11 traceroute to 10.51.66.11 (10.51.66.11), 30 hops max, 40 byte packets 1 (10.52.0.44) 1.436 ms 1.274 ms 1.247 ms DI-DC0 Csico 4900M Default MTU 2 (10.52.0.170) 1.212 ms 1.191 ms 1.161 ms eordsc-di-dmz1-gw cisco WS-C6504-E Te1/4 1500 MTU Te1/5 1500 MTU 3 (10.51.66.11) 3.109 ms 3.091 ms 3.049 ms [root@eordsc-di0a-lb:Active] config # traceroute 10.51.66.12 traceroute to 10.51.66.12 (10.51.66.12), 30 hops max, 40 byte packets 1 (10.52.0.44) 1.102 ms 1.037 ms 1.022 ms 2 (10.52.0.169) 0.933 ms 1.923 ms 1.876 ms 3 (10.51.66.12) 0.836 ms 0.823 ms 0.776 ms

F5 Ticket Number
Ticket # 1-274858331 Place to upload the captures to f5 Use winscp to upload the files Dropbox.f5.com User: 1-274858331

Pass: ricardo.gutierrezo@xerox.com WebEx: https://acs-inc.webex.com/acsinc/e.php?AT=MI&EventID=189524507&UID=0&PW=NYzA5NTdkYmMx&RT=MiMxMQ%3D%3D

Captures
14:19 - 14:24 Agust 14th Port 1442 and server 10.51.66.11

Call Came in at 11:59am August 16th 2013 NO errors were found Started capture at 13:22 for 30 mins August 16th 2013 stopped at around 13:45 Capture a new Ips 10.51.66.25 10.51.66.26 VIP 10.52.24.68 tcpdump -ni 0.0:nnn host 10.52.24.68 or 10.51.66.25 or 10.51.66.26 and port 443 -w /var/tmp/1-274858283_ssl.pcap Client IP 153.8.77.31 (California) (

Bypass Clients
172.16.162.200 F5 bypass Client going to 10.51.66.11

Notes From Alan Rosa

We will definitely need to focus on the performance captures and data coming out of the F5 1600 that supports the composite. Although the memory leak data is somewhat anecdotal, its strong enough not to be dismissed and when paired with the wireshark data, that F5 is dropping sessions for some reason. I would like the following focuses in place with a report back to me on performance at 12:00pm Noon EDT and 5pm EDT. What we need to focus on: 1. Performance on the F5 box memory, CPU, interface performance, etc. 2. Release Mgmt assessment where are we with micro-code and where should we be. 3. Data performance are we hitting any capacity thresholds. The report given back should specifically speak to the questions I have posed above. This may be having significant impact on the SBC application and the 'white screen' issue so we need two teams fired up: one goes full focus on QIP and the second team on this F5. Jason please set up the touch points hopefully 15-20 minutes each time should do it.

F5 Troubleshooting
b tcp stats show 1. Client IP = 172.16.162.151 2. F5 VIP = 10.52.24.132 eordsc-di0a-lb S/N f5-lgha-ypxh Ticket #: 1-274-858-331 1-274-858283 Edgar opened this one a. Pool name: sbcprod_https i. SBC Server IP = 10.51.66.11 0050.5683.4110

ii. SBC Server IP = 10.51.66.12 0050.5683.4112

3. F5 IP to Backend = 10.52.27.23 SNAT Pool

10.52.27.1 10.52.27.2 10.52.27.3 10.52.27.4 10.52.27.5 10.52.27.6 10.52.27.7 10.52.27.8 10.52.27.9 10.52.27.10 10.52.27.11 10.52.27.12 10.52.27.13 10.52.27.14 10.52.27.15 10.52.27.16 10.52.27.17 10.52.27.18 10.52.27.19 10.52.27.20 10.52.27.21 10.52.27.22 10.52.27.23 10.52.27.24 10.52.27.25 Xiangdong Qu | Enterprise Account Engineer F5 Networks _www.f5.com

--------------------------F5 Support--------------------------b tcp stats show Statistics on the TCP packets b interface show interface status TCPdump on F5 tcpdump -ni 0.0:nnn -s0 host 10.52.24.132 tcpdump -ni 0.0:nnn -s0 host 10.52.24.132 or 10.51.66.11 or 10.51.66.12 and port 443 -w /var/tmp/1-274858283_ssl.pcap

tcpdump -ni 0.0:nnn host 10.52.24.132 or 10.51.66.11 or 10.51.66.12 and port 443 -w /var/tmp/1-274858283_ssl.pcap tcpdump -ni 0.0:nnn -s0 <host IP> -w /var/tmp/1-274858283_ssl.pcap from Xiangdong Qu (privately): you can use some filter to replace ,l [root@eordsc-di0a-lb:Active] config # b interface show all INTERFACE 1.1 00:01:D7:B0:ED:86 ENABLED up MTU 1500 | media auto (1000baseT full) trunk: External-Transport aggregation: attached | flow control tx rx (tx rx) | (pkts,bits,errs) in = (587.1M, 3.025T, 0), out = (595.2M, 3.166T, 0) | (drop,mcast) in = (1, 76566), out = (11, 2064) coll = 0 INTERFACE 1.2 00:01:D7:B0:ED:87 ENABLED up MTU 1500 | media auto (1000baseT full) trunk: External-Transport aggregation: attached | flow control tx rx (tx rx) | (pkts,bits,errs) in = (617.3M, 3.263T, 0), out = (583.8M, 3.052T, 0) | (drop,mcast) in = (0, 3375), out = (3, 1930) coll = 0