Steve Hayden's WEA L2 Performance Discussions

WEA L2 Performance Discussions

Attendees
Discussion Points
Action Items
Additional Reference Information

Steve Hayden's Original Correspondence

Continuing Discussions
DSB Sync Information
DSB Performance PMR

The following sections "For DSB" and "DSB/Generic WEA Performance Questions" were discussion points initially tabled by Steve Hayden at the beginning of the teleconference, the section "Items Discussed In Teleconf" were dialog during teleconf. Finally, " Action Items " were discussed/created.

Discussion Points
These are notes/minutes from the 2/5/04 Teleconference/Discussion between WEA L2 support, representatives from DB2e, DMS (invited, no attendee), WEA Development, and the Pervasive/WEA Performance Teams. Everyone should all have seen the correspondence for this meeting describing the situation and need for this meeting. I have a set of perhaps naive questions, not sure if all of them should be asked in this forum, but I think we need the answers to.

For DSB:

Our current issue is DSB WEA rampup for their application using WEA/DMS/DB2e and how do we position/prepare ourselves (WEA L2, and who else?) to assist/react to them. Additional discussions need to be had (in this teleconf or another) relative to other components (ESS, INS, LAS, etc.).

IBM Denmark was to have had a meeting Wednesday (2/4/04) to discuss their role in DSB rampup (Their results were to be sent to me).

1) Can we offer recommendations for the device (PPC) characteristics (model, memory, communications, etc.?)

2) Can we offer recommendations for the server(s) characteristics (distributed, model, memory, communications, etc.?

3) What Subject Matter Expert(ise) can/should be offered by what organizations within US. (PIM/email)

3) What WEA performance characteristics for DMS/DB2e exist to help IBM/DSB to plan architecture for 1300 users.

4) Confirm ownership of WEA performance issues/assistance.

Notes about DSB: According to Barry Beggs (WEA Development), DSB does not want AIX (can/should we convince them otherwise).

DSB/Generic WEA Performance Questions

Is WEA Performance team to be all encompassing (I should not have to go to individual component team(s) for perf issues)?

From a development perspective, does WEA Performance team have a "design/build it and we will measure it", or "let me help you design it" philosophy/behaviour? (need to have a services representative from somewhere in AMEA to be on the ground at DSB to be working the issues through the team).

Who has the knowledge to aid in design/architecting DSB (or any large scale user) solution. "Who is in charge to test"?

Who's job is it to aid in design/architecting DSB (or any large scale user) solution.

What is process/role for WEA L2 "DSB" request for performance "engagement".

What is process/role for WEA L2 "Generic" request for performance "engagement".

Curtis - start with building blocks, what are the performance characteristics. On one box, and then in a distributed system. Curtis knows HOW, but do not know the characteristics of scaling. We do not know what the characteristics are to determine if we need to add another box. Focus has been on ESS, DMS is one component we have not focused on.

Barry pointed out we have some numbers for DB2e and DMS. (single node only.) What is the capacity of adding additional servers. What is the optimimum/minimum requirements. Gets complex as a bundle. What is the end result to present to DSB. We need to do testing/analysis to determine what to say to DSB. Need to line up resources and commitment to address this issue.

Discussion of WEA 5.0 and how it fit's into the picture. Will it be there in time? Key factors "what kind of performance and capacity would be provided by WEA 5.0?"

Need IBM Denmark to engage customer to determine if DSB schedule is realistic and whether it fits with delivery of WEA 5.0.

Who owns this issue? L2 is looking for a performance owner. L2 is the focal point for driving the issues, communicating the DSB problems to development to get guidance on problems.

L2 normally does not own performance. In WEA will have different role. Discussed notion of a "Swat Team".

Luis "from feedback from customer, the kind of testing they have been doing is not that realistic. Using different HW for testing and production. Someone has to involve the customer and test the loads and HW before they go to production. DSB is testing TODAY, and within short amount of time will have real data. (DSB test vs production servers/PPC's is an issue).

Need DSB to provide data on user behavior characteristics?

Louis - need someone/team responsible of integrated/performance testing and collecting data.

Curtis stressed the gap. Testing is done at the component level, we dont stress at integrated level. It's only end to end. We dont do "Live client/device concurrent testing". We only use simulators.

Action Items

Bruce Morris is to initiate discussions related to a Swat Team

Steve will contact IBM Denmark about their discussions.

Need to have follow on. Need to have management team discussion. Need to plan for future. Wait for 5.0?.

Next Teleconf at 10:00 Wednesday (2/11/04).

Attendees

Steve Hayden (WEA L2)
Bruce Morris (WEA L2)
Derrick Weaver (WEA L2)
Louis Mau (DB2e performance?)
Kirk McKinlay (WEA Performance?)
Billy Capps (WEA Development)
Curtis Ebbs (WEA Design/Integration)

("Tentative")
Jay Tunkel (PvC Perf. and UCD)
Barry Beggs (WEA Dev)

(No reply as of 2/4/04 @ 6:00pm)
Bill Siddall/Raleigh/IBM@IBMUS
May Wone/San Jose/IBM@IBMUS
Norbert Runge/Santa Teresa/IBM@IBMUS

Additional Reference Information

Bill Siddall's Comments (added Feb 5 @ 5:30)
Bo Pedersen's IBM Denmark Discussions

Bill Siddall's Comments

We have quite a bit of performance data for DMS in a standalone environment. That is, when not integrated into WEA. Joe Carter is the person who has done the performance work on DMS.
As we have seen with DSB, the performance of a component can change quite a bit when integrated into WEA and when other components are in the mix. In DSB, it was the additional overhead associated with user services to WPS to AD.
Our performance testing to date has used high-speed links and simulated devices. This is done to measure server side performance. In DSB we have seen that the device may be the bottle neck. I don't think link performance has been an issue (yet) at DSB, but when a customer does syncs over GPRS, that may become an issue.

Bo Pedersen's IBM Denmark Discussions

(note that IBM Denmark's discussion minutes were to be distributed by Henrik Saaby, and have not been received. These were "highlights" from Bo)

We discussed the activities that will take place during the next month in the project.
Headlines were:
Software:
    Planned GA for next release of WEA
    Which advantage can DSB expect
    Which implications can DSB except
    When should we advise DSB to migrate

Infrastructure:
    DSB requirements for availability and performance
    DSB skills
    Pitfalls and risks

Support:
    WEA product support
    Local ITS support

Steve Hayden's Original Correspondence:

Greetings,

This correspondence is to request your and your appropriate team members (or appropriate delegates) participation in a teleconference to discuss DSB (Danish Railways, an IBM "reference" customer) WEA implementation performance and their ramp-up. This teleconference is scheduled for Thursday, Feb 5 @ 9:00AM.

DSB has just finished completion of a "Stress testing" phase of their rollout of the WEA product. Their stress testing included simultaneous synchronization of 30 devices, which includes an initial sync of DMS and DB2e (approx 27Mbytes of applications/data), and subsequent DMS/DB2e synchs with varying amounts of DB rows/data. Considerable amount of IBM resources and differing expertise were involved solving problems (PMRs) and performing performance tuning which, although did not meet DSB's initial desired performance levels, were deemed "acceptable" by DSB.

They are currently training several "super users" (20 users, which include 11 train conductors and 8-9 sales people) on the use of this product with intentions of going into production on Thursday of this week (2/5/04) (real conductors, real trains, real tickets, real money!). DSB has an aggressive ramp-up scheduled, with 100-200 users being added at time up to a quantity of 1300 users by September 1, 2004.

Their current environment is Windows 2000, and as alluded to before, has had considerable IBM energies expended to get them to this point. NOW, we must begin to work with them to determine their architecture/environment and support considerations for ramping up to this 1300 users.

Thus the need for your involvement in this teleconference. We need to understand and be able to make sound recommendations to DSB relative to their requirements for their environment to be able to provide the required performance to handle this number of users, as well as assure we have the required internal resources/processes prepared to most expeditiously handle issues related to their environment. Although DSB has provided us with the number of expected users, they have not provided official numbers or "guesstimates" of the number of concurrent users synchronizing, amount of data, anticipated usage patterns, etc. for their applications.

HOWEVER, we do know that there will be lots of users and lots of data, and we must assure we can meet (or exceed) their requirements/expectations, with appropriate processes in place to provide responsive on-going support to DSB (and future customers). DSB will likely require a distributed, scalable, clustered, highly available system, to meet their performance requirements, which could provide IBM a model for other customers (DSB in the future intends to implement ESS also).

Your participation would be greatly appreciated.

Teleconf info:

Dial-in numbers:
   Toll Free: 1-800-493-7360
   Toll: 1-719-219-0041
   Tie Line: 650-3206
Passcode: 495926

Regards,
Steve Hayden   (shayden@us.ibm.com)
Phone 919-543-7375 Tie 441-7375
IBM Certified for e-business - Solutions Technologist
IBM Certified Systems Expert - Admin. for WAS AE 4.0

Continuing Discussions

From Barry Beggs 03/02/2004 @ 04:09 PM ,
Need some info to help us. As we discussed earlier, will DSB be limited to 30 access points, meaning 30 concurrent syncs for the immediate future. If its 30, then I think we are scaling right now on the production machine; need DSB verification. If we are scaling with 30, then we need to talk to verify the point products, db2e and DM will scale to a the number we suspect in the WEA4.3.1 timeframe (Curtis brought up a good point about point product pruning issues for the DB and filesystem that need to be discussed) . Two wildcards are when they want to go use ESS and PPC2003. There will be some obvious performance issues beyond the current setup for ESS.

Can you follow up on these:

DSB
- How long for the 30 access points?
- Is the production machine scaling to the 30 concurrent syncs?
- When will they start using ESS?

DB2e/DM/ESS?
- Are there any scalability issues like pruning
- Any limits like the 1000 user one we know of (say MDAC or other integration points with WEA)

Maybe we can put off addressing some of the big scalability issues until WEA5, if we are lucky. I assume there is work going on in WEA5 that will cover DM and db2e better than we have in WEA 4.3.1.

From Barry Beggs 03/12/2004 @ 02:33 PM
Steve, Curtis gave some good suggestions about how to handle DSB capacity monitoring. We have them in production based on the 30 access points. DSB should be monitoring their capacity to make sure it stays in parameters while they scale users. Suggestion is to us Windows performance monitoring and use some reports to get an idea of when the syncs are going on. Can you initiate this discussion with DSB. We can also use some of the information to feed back into Curtis' performance models. Thanks --Barry

From Stefan K to Barry 3/15/04 @ 3:50AM,
Please can you supply any "performance model" information from Curtis - I'd like to be kept in the loop.

I used WebSphere Resource Analyzer a lot when at DSB.

I found that Windows Performance Monitor tool could not always log all the items I wanted - please see my performance tuning doc - this outlines some important items to monitor with Performance Monitor (it now may work due to more RAM being available in the DSB machines).

From Barry Beggs 03/15/2004 @ 09:46 AM

Hey Stefan, Curtis can reply to the other q's. What set of parms should we be monitoring on the machine?

From Stefan J Kwiatkowski 03/17/2004 @ 07:53 AM
Some things to look at - obtained from various redbooks, Tivoli Performace doc for WEA 4.2, DB2 Performance doc :- see table "Data to Monitor For DSB"

Discussions With Curtis Hrischuk/Kirk McKinlay 3/25/04

Curtis "IGS should be involved". Writing a PMR does not scale. Who needs to put the process in place. Curtis is to task his manager
Jay Tunkle to get process in place for this type of engagement. Bruce Morris will/should be involved.

Need to understand the usage patterns and workloads of DSB. Curtis/Kirk are to look at chart from Stefan to confirm this is what should be monitored. We are to get back together in 2 weeks to synchronize our activities.

4/12/04

Again discussed need for process. Chris to discuss with Jay.

For Data to monitor "IO" (Windows "PerfMon"). Need the model for DB2e performance (Steve will approach DB2e for performance model stuff). (For future need INS, DMS, LAS performance in context

Time of day and type of synchronization. How much cost for scenario. Time of day for each type of 3 syncs. Toby Manners is the WEA project manager and should help with this type of engagement (Curtis's group to follow up with Toby).

When is the busy time of day and what resources are being used then and cost of a sync. Need patterns of usage. Who syncs what when?

Data to Monitor for DSB

Performance Object	Counter	Instances
IBM HTTP Server	KeepAlive connections	All
IBM HTTP Server	Waiting for connection	All
	Total kilobytes sent	All
	Total request/seconds	All
Processor	% Processor Time	All
Memory	Available Kbytes	All
DB2 Database Manager	Current Number of connections for DB2 Connect	DB2
	Number of MAX_AGENTS overflows	DB2
	Number of connections waiting for host reply	DB2
	Number of connections waiting for client to send request	DB2
	Remote Connections	DB2
DB2 Databases	Exclusive Lock Escalations	DMS/DSYCTLDB plus Mirror database for DB2e
	Lock Escalations	DMS/DSYCTLDB plus Mirror database for DB2e
	Lock Waits	DMS/DSYCTLDB plus Mirror database for DB2e
	Maximum number of concurrent connections	DMS/DSYCTLDB plus Mirror database for DB2e
	Number of Lock timeouts	DMS/DSYCTLDB plus Mirror database for DB2e
	Sort overflows
	Update/del/ …. inserted
	Buffer pool …..

From Bo Pedersen & DSB 03/18/2004 @ 04:28 AM
Amount of data + sync’ing from device to server.

DSB Sync Information

Estimated size of THTPRD (Production DB)

Table	Avg. size pr. row	No of rows	Period to keep	Table Size	Index Size	Total size
Bruger (user)	50	1300	Always	63.5Kb	10.2Kb	73.7Kb
Maskine (machine)	145	2600	Always	428Kb	193.2 Kb	621.2Kb
Salgstur (salestrip)	90	1300*200	1 Year	26 Mb	8.5 Mb	34.5Mb
Ordre(1) (Order)	3095	13002014	14 days	1.22 GB	53Mb	1.23Gb
Ordre(2)(Order)	3095	13002030	1 month	2.6Gb	114.3Mb	2.7Gb
Log(1)	4211	1300*30	1 month	186Mb	60Mb	246Mb
Log(2)	4211	1300*14	14 days	87 Mb	30Mb	117Mb


Tht_billetnr (Tht ticket no.)	29	700	8 days	20Kb	10Kb	30Kb
Tht_billetordrer (Tht ticketorder)	239	700	8 days	163.4Kb	80Kb	243.4Kb
Tht_dagensnr (Tht daynumber)	29	7	8 days	10Kb	5K	15Kb
Tht_pladsordrer (Tht seat order)	424	700	8 days	290Kb	140Kb	430Kb

Total after 14 days data in ordre and log				1463Mb	92Mb	1555Mb
Total after 14 days data in ordre and log				2714Mb	183.3Mb	2897Mb

Space has been allocated for the tht tables and the housekeeping programs are implemented and running today.

For the tabels bruger, maskine, salgstur, ordre and log we need to find the requirements for size and implement the housekeeping procedures.

Estimated amount of data sync’ed between server and device

Table	Server <- -> device	Frequency	Number of rows	Amount of data
Bruger (User)	->	1st time	1	50b
Maskine(Machine)	<- ->	1st time each way with new machine	2	390b
Salgstur (salestrip)	<- ->	At every synchronization an item is exchanged	2	180b
Ordre (Order)	<-	Every sync	Avg. 20 - can be up to 100++	61900b
Log	<-	Every sync	Avg. 1	4211b

Tht_billetnr (Tht ticketnumber)	->	Each day	120	3480b
Tht_billetordre (Tht ticketorder)	->	Each day	120	28680b
Tht_dagensnr (Tht day number)	->	Each day	2	25b
Tht_pladsordre (Tht seat number)	->	Each day	120	34800b
Total Avg.				130.6K
Total max (30 orders + 30 logs)				552K

DSB Syncs Per Day

	Max concurrent syncs	Time of day	Average amount of data to transfer for DB2	Average amount of data for other DMS	Average amount of data for others	Average amount of records for DB2	How many initial sync per day in average
Phase 1 jackets alone after all users are educated	30	From 04:00 AM to 02:00 AM	130.6 KB pr user pr day (max 552 KB)		N/A	387 rows per day per user (Max 500)	5
Phase 2 WLAN on selected stations and jackets active	Approx 50	From 04:00 AM to 02:00 AM	130.6 KB pr user pr day (max 552 KB)		N/A	387 rows per day per user (Max 500)	5
Phase 2 WLAN on selected stations - WLAN in IC4 trains and jackets active	Approx 60	From 04:00 AM to 02:00 AM	130.6 KB pr user pr day (max 552 KB)		N/A	387 rows per day per user (Max 500)	5

DSB Performance PMR (02938,060,678)
"Request for doc., when WEA solutions are scaled"

-PEDERSEN, BO        SC-5724C9402 -L41S/DKECSW-P3S3-04/02/23-14:02--CR
Problem Details
Abstract:
ESR: Request for documentation, when WEA solutions are scaled to +1000 user.
Operating System: WIN 2000 Server
Product or Service: WEBSPHERE EVERYPLACE ACCESS PROCESSOR
Component ID: 5724C9402
Product Version/Release: WEA 4.3

Request for guidance and documentation, for WEA solutions that are scaled to +1000 users. DSB needs knowledge of the consequences and performance impact, when more that 1000 users are registered as users.

More details regarding DSB requirements, covering total number of users, number of concurrent users, amount of data, DB2 load etc will be add to the pmr as soon as we have the data from DSB.

Thanks
Bo Pedersen

+HAYDEN, STEVE         -5724C9402 -L13G/WPVC -P3S3-04/02/23-16:38--CT
S5> SERVICE GIVEN= 99 SG/99/
CUSTOMER REP:
--Ole Prebensen
PROBLEM:
--Request for WEA Performance and rampup assistance
ACTION TAKEN:
--Began discussions with WEA Performance and Development team, and notified customer of such.
ACTION PLAN:
--FUP with WEA Dev/Perf and customer by 3/2/04


+HAYDEN, STEVE         -5724C9400 -L13G/-------P3S3-04/02/27-19:54--AT
ACTION TAKEN:
--Discussed with customer. Requested additional inflormation (number of users, amount of tickets/orders, usageage patterns, etc). BPedersen is
to discuss this with customer and provide results.
ACTION PLAN:
--Use this PMR to drive WEA Development and performance to prepare for engagement with customer to provide guidance for rampup.

+HAYDEN, STEVE         -5724C9400 -L13G/-------P3S3-04/02/27-19:54--AL
ACTION TAKEN:
--Discussed PMR with customer. Requested WEA L2 Mgt and WEA Performance team determine requirement and mechanism for engagement for customer rampup to full production
ACTION PLAN:
--FUP with WEA L2 Mgt 3/9/04 for discussion and disposition of PMR for FUP with customer on 3/10/04


+HAYDEN, STEVE         -5724C9400 -L13G/PVCBE -P3S3-04/03/17-18:29--CC
S5> SERVICE GIVEN= 99 SG/99/
ACTION TAKEN:
--Discussed with customer who is to provide translated version of data (Danish to English) for information provided previously.
ACTION PLAN:
--FUP with WEA Performance team to determine what values can be set to enhance performance and what should in customer's environment should be
monitored to provide indication of robustness and requirements for scaling. Will FUP when English version of data is provided or by 3/24/04.

+HAYDEN, STEVE         -5724C9400 -L13G/PVCBE -P4S3-04/03/24-17:40--CT
ACTION TAKEN:
--Received English translation of information from DSB. Scheduled meeting to discuss this PMR and performance with WEA Performance team
(CHrischuk) for tomorrow.
ACTION PLAN:
--Meet with WEA performance, participate in on-going dialog with customer to resolution of PMR for DSB rampup. Will FUP with customer
3/31/04 regarding status.

+HAYDEN, STEVE         -5724C9400 -L13G/PVCBE -P3S3-04/03/31-14:50--CC
ACTION TAKEN:
--DSB did not participate in telconf call today. Had discussions with WEA Performance team (CHrischuk and KMcKinlay). They are to analyize the
data from DSB and make recommendations on what data elements to monitor in DSB's environment for visibility for when DSB might need to prepare
to scale.
ACTION PLAN:
--FUP with WEA Performance team 4/8/04 and determine what should be commuinicated to customer.

+HAYDEN, STEVE         -5724C9400 -L13G/PVCBE -P3S3-04/04/08-17:00--CC
ACTION TAKEN:
--WEA Performance personnel (CHrischuk) had to postpone teleconf to discuss DSB rampup. Rescheduled for 4/10/04.
ACTION PLAN:
--FUP with Performance team 4/12/04

Teleconf 2/27/04

Attendees: Stefan Kwiatkowski, Curtis Ebbs, Roberto De Armas, Joe Carter, Kirk McKinlay, May Wone, Chris Martinez.

Joe Carter WEA 4.2 performance had 5000 users no syncing. Joe is the only person doing DMS performance. During 4.2 (DMS 1.3, same as WEA 4.3.1). "We do not know how the components will interact together." Things to look at were able sizes and buffering. (normal DB2 tuning stuff).

PVC Performance has to this point PIM and email. Have not done other components. Scalability testing is only on ESS.

DSB has AD to consider also.