You should be able to run both SPs up to 70%, althoughI like the 50% rule better. If 1 SP somehow fails (or reboots during an upgrade), the remaining SP needs to be able to take over anything the other SP was doing. Officially it's 70% on both, at the same time, but if you actually reach that level, I'd be alarmed.
The first thing that I notice is the 2500 IOps for flash. The latest RoT says it's 3500 for FAST VP flash and 5000 for FAST Cache flash.
Then in your 4th column, you correctly state the number of total IOps for each tier.
If you have 100% reads, the number shown there is what that tier can accomplish.
For writes the RAID level determines the write penaly:
For RAID 10 the penalty is 2
For RAID 4 the penalty is 4
For RAID 6 the penalty is 6
So if your NL-SAS pool can handle 3500 IOps and it's configured as RAID 6, with 100% writes the number of host IOps will be 3500 / 6 (write penalty) = 583 host writes per second
With your 15k SAS pool in RAID 5 the number is better: 14400 / 4 (write penalty) = 3600 if doing 100% writes
And with FAST VP flash (that's 10x3500) 35000 / 4 = 8750 if doing 100% writes
In real life it will be a mix of reads and writes. So suppose it's 50/50 reads and writes and you're using the SAS pool and your hosts are doing 5000 IOps, it's 2500 for reads and 2500 for writes, which translates into 2500 + (4x2500) = 12500 IOps.
Ur solution is very helpful, how does total IOPS distributed when there are mixed drives and mixed RAID in a single storage Pool, for example in a single storage pool we have 10 SAS flash Vp, 10 SAS, 10 NL sas,
Adding up all the disk IOPS as per RoT, we get approximately 37600 IOPS
Single storage Pool with mixed drives and mixed RAID (5&6)
if the pool can handle 37600 IOps, it's the total of read AND write, so if you're doing 10000 read IOps, that leaves 27600 for writes.
If these 27600 are 100% RAID5, then that means it's 27600/4 host IOps. If it's 100% on RAID6, that's 27600/6 host IOps. If it's 50/50 as seen by the storage, so 50% to RAID5 and 50% to RAID6, 27600/2 = 13800 for RAID5 and 13800 for RAID6. That means 13800/4 = 3450 host IOps to RAID5 storage and 13800/6 = 2300 host IOps to RAID6 storage.
10000 read IOps
3450 write IOps to RAID5
2300 write IOps to RAID6
10000 + 3450x4 + 2300x6 = 37600.
BUT: since you don't know where every 256 MiB block resides of a random LUN in your pool, you still can hit slow storage when you're in need of faster storage. And that's where FAST Cache comes in: it absorbes writes to flash instead of SAS or NL-SAS and it places blocks in cache when accessed more than 3 times (I think it was 3).
So your pool can do 37600 IOps, but that doesn't mean every block in every LUN has that performance. It's the total of the pool and assuming the hot spots never move, these are placed in flash and the almost dead data resides in NL-SAS. If you see a lot of data movement every day, that might indicate that your flash or SAS tier is too small.