SAP System Cloning & Copy, the painful post processing tasks and why it takes so long?
A couple of months ago, I started a discussion on the fact that every SAP customer needs to make a copy their SAP Production system in this blog post, and I mentioned that more often than not, SAP customers need to perform quite a bit of post cloning activities before the copy of the system can be used for other purposes. Examples of such activities range from changing the instance name from PRD to QAS, to changing the path names for data files, to turning off printers and cancelling current print jobs, to cancelling existing pre-scheduled batch jobs, to redirecting RFC destinations, and so on.
So let’s review what these post-processing tasks are all about and why they need to be performed.
First, let’s examine when post-processing is NOT needed: if the system copy is made purely for backup purposes, nothing should be changed since we want to be able to restore this copy of Production back exactly as it was copied. This copy of the Production database could be then sent to tapes for a more permanent backup, or it could be used as a point-in-time copy of a steady state, where the database can be easily and quickly mounted for a very aggressive RTO and if needed, the database can further be recovered by applying redo logs for a short RPO.
But if the system copy is made for other purposes, such as to obtain a fresh copy of the Production database for new development work, or to create of a sandbox for testing new patches or version, or to have a functional or load testing environment using the latest Production data, or to create a training environment, then the post processing work must be done.
Now why is that? Well for one thing, you don’t want to have 2 SAP instances named PRD in your landscape and network environment, as you cannot even begin to imagine the havoc that such an unfortunate event will create – this is very basic SAP Basis 101 that every SAP Basis person knows by instinct since it was taught from day one.
So at the very least, the SAP/database SID name must be changed, say from PRD to QAS, and the entire OS level path name for the database executables and datafiles must also be changed since SAP had unfortunately hard coded the SID name into such paths, for example \data\oracle\PRD\sapdata1 needs to become \data\oracle\QAS\sapdata1. If the path names are not changed, Oracle (or the database being used) will have trouble starting correctly.
Most SAP customers have long ago automated this task by using a combination of shell and SQL scripts, and the level of effectiveness depends greatly on the skill of the script writer. Needless to say, the maintenance of these scripts is not a trivial task, especially if the script writer leaves the company and did not leave behind good documentation – talking about a potential real nightmare!
But that’s merely the BEGINNING of the significant amount of work which needs to be done. This is because the SAP Production system has printers configured and there may in fact be print jobs in the queue when the system copy was made – we don’t want these print jobs to be sent to the Production printers, especially if that printer is located half way around the world, so the print queue must cleared. The SAP Production system also has RFC destinations defined and we definitely do not want that million dollars purchase order to be sent out twice via EDI (the real one was sent from Production and the bogus one was sent out by the Production clone when it came online). Again, the kind of havoc that this sort of carelessness creates will probably cause someone to lose their job!
This is why the post cloning work takes so much time, since after the system copy is done and the SID and path rename has been done with the scripts, a skilled SAP Basis person must log into the target SAP system through the SAP GUI and manually execute the post-cloning tasks. Just waiting for that skilled SAP Basis person to become available in order to perform these tasks could take days, unless this person drops something else to do this work because this is a rush job and there is a crisis at hand.
And we’re not talking about a few tasks here. Oh no! The list of post-cloning tasks needed to be executed varies from customer to customer, but the list is so long and people do not want to forget any step that such a list is often maintained in an Excel spreadsheet, and it is part of the standard Run Book.
Just for fun, here is a partial list of post-cloning tasks for SAP R/3 or ECC from one of our customers:
1. Set Oracle parameters for non-productive operation
2. Implement OSS NOTE 185821 (this task has substeps describing what needs to be done)
3. Implement OSS NOTE 641630 for missing PLAN_TABLE
4. Verify database logging
5. Grant dba to OPS$<SID>ADM and OPS$ORA<SID>
6. Truncate monitor tables
7. Truncate load tables
8. Update the database statistics
9. Connect via R3trans –d and start SAP
10. Set SAP parameters for productive operations
11. Start SAP (disable background processing)
12. Transaction SICK (SM28) to verify that the system is stable
13. Install the SAP license (SLICENSE)
14. Configure the Workbench Organiser (SE06)
15. Configure the Transport System (STMS)
16. Run report RDDNEWPP (as user DDIC)
17. RSBTCDEL to delete all existing job logs (SE38).
18. RSPO0041 to delete all spool files (SE38).
19. Clean-up TEMSE (SP12) to clear out and cancel all existing print jobs
20. Adapt RFC destinations (SM54, SM55 and SM59)
21. Adapt SAPOSCOL destination (AL15) since this system has a different SID name
22. Import default, instance and start profiles (RZ10) for non-productive systems
23. Create instance for operation modes (RZ04) (RZ03) for non-productive systems
24. Re-fresh logon groups (SMLG / RZ12)
25. Generate ABAP loads (SGEN) to speed up execution of standard SAP Tcodes
26. Re-assign printers (SPAD) away from the Production printer pools
27. Run report CSM_TAB_CLEAN (SE38)
28. Run report SDB3FORA (SE38)
29. Run report RSSNAPDL (SE38)
30. Run the database checks (DB02)
31. Check SAP/Oracle memory allocation (ST02 / ST04) for non-productive systems
32. Check SAP processes (SM51 / SM66) for non-productive systems
33. Verify printer installation (SPAD) are pointed to non-productive printer pools
34. Verify background environment (SM65) for non-productive systems
35. Verify update and locking (SM13) (SM12)
36. Verify the NLS configuration (SE38 --> RSCPINST) (OSS NOTE 42305)
37. Configure system administration jobs (SM36)
38. Execute transaction RSSGPCLA
I have seen some SAP system copy post-processing lists run into hundreds of tasks.
It’s important to note that as far as I know, the majority of SAP system copy or cloning solutions in the market today DOES NOT handle this post-processing work. EMC is the only vendor which offers Intelligent SAP Cloning Solutions for ABAP and Java that incorporate post-processing activities for a true end-to-end SAP system copy - that’s why we refer to our solutions as “Intelligent” because once the post cloning tasks have been analyzed, configured, tested and successfully implemented, you can in fact have 70% to 80% automation (and in some use cases, 100% automation) of the SAP system copy process.
In most use cases, EMC’s Intelligent Cloning Solutions do NOT require that a SAP Basis person log in to execute these tasks, because our orchestration software, Replication Manager, can automatically call these post-processing scripts to execute as many tasks as needed in practically any customer use case. Replication Manager also handles the renaming of the SID and the data OS level path names out of the box, without the need to build any script. Better yet, Replication Manager stores all the necessary login and password credentials needed to perform these tasks in its own internal database, and therefore such sensitive information as the Oracle SYS password is not being sent in clear text over the wire as would be in the case of shell scripts.
OK, so we’re able to automate the SID and path rename, along with the needed post-processing activities, but in some use cases, we’re not out of the woods yet, because of a dreaded SAP script called BDLS. What is this monster?
In its infinite wisdom, SAP hard-coded the SID name (such as PRD) into the tens of thousands of database tables in a typical SAP system. If the newly cloned system must communicate with other non-Production systems in the SAP landscape for testing of IPCs or RFCs (such as when SAP XI or MDM is involved as an example), then all these hard coded SID names must also be changed, and SAP supplies its customers with the BDLS script to handle such a chore.
Now just imagine any script navigating through terabytes of data, at the database table’s level, to perform a search and replace of PRD to QAS one table at a time – how long do you think that such a script would need to run?
The sad answer is that BDLS can take days to run! At the typical SAP users conferences, customers often get together and trade tips and tricks on how to speed up BDLS, and I have witnessed more than a few high fives when someone found a way, thanks to a tip from a peer, to reduce their BDLS processing from 30 hours to 8 hours as an example. Many skilled SAP consultants have made nice money helping customers reduce their BDLS processing time! In today’s SAP environments of multi-terabyte databases, any BDLS run of under 4 hours should be considered a triumph.
OK, now that the bad news is out of the way, I have good news: all EMC intelligent SAP cloning solutions using Replication Manager can call the dreaded BDLS script as part of the post-processing work.
Now you can begin to see why quite a few SAP customers can only afford to do system copy or refresh once or twice a year, as the effort involved is too great!
So at EMC, we know this challenging scenario all too well, since at one time or another, all of us in the EMC SAP Solutions Group were SAP Basis consultants or admins, and we have had to endure this pain ourselves.
That’s why we built mutiple SAP Intelligent Cloning solutions to suit a wide range of use cases, from the most simple and basic system cloning to create a Sandbox or a quick test environment which will be quickly thrown away, to the most comprehensive and fully SAP DDIC-integrated cloning solution using Cisco’s Tidal Intelligent Automation software. Above all, we offer real choices to you, our customers.
In future blog posts, I will go deeper into the weeds to explain and contrast how the EMC Intelligent SAP Cloning Solutions work, what SAP cloning use cases would benefit from Tidal vs simple SQL scripting, and which use cases would make the most sense with the innovative TimeFinder SNAP functionality available only on our market leading VMAX with Enginuity 5875.
More to come…
Tim K. Nguyen
SAP Global Technical Evangelist & EMC Cloud Architect
EMC Solutions Group