Done – All 5!

AWS all

So I got them, all 5 of them, the top 4 on this list in less than 3 months. But don’t let the limited time fool you, I spent vast amounts of time, every evening and over 30 hours on weekends, stealing from personal, family and sleep time. I wasn’t intending on doing the Devops Pro till the end of July, but business reasons “pushed” it quite a bit forward on my schedule.

In retrospect, my advise is plan to take all associate ones together, as closely scheduled as possible, maybe a couple weeks apart on each. There are 2 main reasons for this. First of all, a lot of material is overlapping, and even the material that isn’t helps towards passing the other ones. Second, the material and knowledge attained in all three as a whole helps immensely at any one of the pro exams. The order in difficulty, you will find this in blogs quite a lot and I agree 100%, would be developer, architect, sysop from easiest to hardest.

For the sysops exam I followed the same method as discussed in the blog posts below. I did both and linux academy courses and used my free tier services as much as possible. That’s my second piece of advise. USE the free tier and don’t be afraid to get into small charges also. Those 5 tests combined reference a very large set of technologies, many of them which have been evolving for a while into separate beasts packed of features and abilities. There is no way to remember these or distinguish between their merits unless you use them. I built several Lamda functions, coming up with business scenarios on my own, even though I knew Lamda had small coverage on the exams. Have fun! – AWS is a true “enabler” giving you options to come up with solutions that needed months of planning and work before. Play, scratch and destroy, play again!

Now the Devops Pro Exam. This for me was far harder than the CSA pro exam. I was surprised that opinions in other blogs seem to vary on this. Maybe it was just my lack of experience in the Devops field and my more heavy experience in architecture. But, the questions seemed longer, “verbier” with many more options. On a few there were answers till (f) I think, by the time I was reading (e) and (f) I had already forgotten what (a) and (b) were about. This was the only test in my life – apart from an advanced Math Statistics course back in college🙂 – that almost none of my clicks were “given” with 100% certainty. This adds to the stress already high by constantly being behind in time. When I reached question 80, I had 45 seconds left on the meter.

There is no course online for the Devops Pro other than the one – but that is a fantastic course!!! Those 2 guys have done a superb job in creating a framework that guides your studies for the exam and in the delivery of it. Watch each lesson at least twice, pay attention to all the details – I think there wasn’t a single word spoken in that course that didn’t somehow help in the exam.

And… play, get a free-tier and play. They provide a github repo with material to build your own Opsworks and Elastic Beanstalk deployments. Branch it, alter the code, deploy and redeploy, go through rolling updates, A/B testing and Blue/Green deployments. You can do amazing work with free-tier and well under a hundred dollars of AWS charges. You need to “feel” the differences between EB, Opsworks and Cloudformation before entering that exam.

Once again, I hope this advise helps you and good luck if you are on the all-5 quest! I didn’t want to repeat advise given below, so if you are just starting read my 3 posts in chronological order. Cheers!

Screen Shot 2016-06-22 at 7.46.04 AM

Cranking on the AWS certs… 3 down, 2 to go!

AWS CSA pro and ADevI should rename the blog “my certs”, since this is all I seem to be writing now days. Quite heavy work and personal schedules, as much as various NDAs and no time to sanitize work, has kept me from posting some really interesting work and case studies.

Anyway, back to the certs🙂. I have managed to pass both the AWS Solutions Architect Professional exam, as well as the Developer Associate Exam the last few weeks!!!

To start with the Developer associate, you will read around that its probably the easiest of the 3 associates exams and I can attest to that – even though I haven’t taken the SysOps one yet (but have seen the material).

I started to use Linux Academy a bit back, to up skill on some technologies and since I had the subscription I went through their Developer course also. Their course is simply superb, especially the quizzes, both in the end of each section as well as the final large one. Of course, I went through the one also – a site which is becoming one of my favorites. If you take both those video courses and spent even a small amount of time familiarizing yourself with these services, this is quite an easy exam to pass.

Now for the big one. A couple of years back I passed the VMware VCAP exams which were brutal. Over 3 hours long, with 3/4 page questions or hands-on on an environment you could easily break or lock yourself out of, especially under the time pressure. I thought I wouldn’t witness a harder certification exam. Well, this one is up there.

Close to 3 hours – 170min – and 80 very long questions, mostly situational with very long answers as well. Time is barely enough to read everything twice and then you have to choose in seconds and move on.

All this along with the fact that many times, 2 or even 3 out of the 4 choices are correct architecturally, and you have to go back to see if it fits better the resiliency, cost effective or other parameter which is stated somewhere in the question. I even think one of my questions stated simply “the most effective solution” – which I never managed to figure out the nature of effectiveness requested🙂

To prepare, I used once again both linux academy and This time though, I think the course is a bit too spartan and lacks detail and depth. Still a great accent and color of voice to listen-to though, easily cruises through, but you would probably need a lot more experience than mine to pass solely on that course. The linux academy one here is a gem. A lot of details, good exam tips, a lot of anti-patterns mentioned in scenario basis/types, as well and once again great quizzes at the end of the sections and a big one at the end. These quizzes were my summary refreshers too, for the last day before the test.

Many people mention this, in blogs and forums, do take their advise seriously and… SLEEP WELL – I am pretty sure I wasn’t even reading straight, let alone thinking straight by the last 30-45′. Once again, very long test, long questions, long answers… you basically speed-read for 170 minutes through complex problems.

Along side the videos, both sites do a great job of pointing to the right whitepapers and FAQs. I, once again, read all the whitepapers twice carefully, but basically only skimmed through the FAQs.

For this one, you need hands on, experience and experimentation. I would rush though, if you are close on preparation, as AWS has released a lot of new services and many changes to existing ones since the summer of 2015  – and all these don’t seem to have made their way in the test yet. Focus on VPC/network/VPN/VIF/Endpoints, STS/Role assume/Federation, multi tier with Autoscaling/Auto-deployment…. these were over 80% of my test I believe.

One last tip, if you use linux academy, go to your preferences, switch to HTML5 from flash. This will give you the ability to watch/listen at 1.5X the speed, which as you’ll see is VERY valuable🙂 Actually, I also run at 1.3X. And, if you ever need a refresher on something, you can run them at 2X, get the main points fast as well as smile/grin at the voice distortions🙂

I am leaning towards taking the Sysops Associate one before I go for the Devops Pro – even thought not needed – as I have seen these training courses fill-in some great bits of info here and there.

Hope I helped to clarify the nature of these exams to anyone interested, good luck!



AWS Certified Solutions Architect – Associate Passed!

Solutions-Architect-AssociateI passed the AWS Certified Solutions Architect – Associate exam this week.

I must admit it was quite tougher than I thought.

I have used AWS for the 2 of the last 3 years but not so much the past few months, and never in a very structured way and mostly concentrated only on the products needed at each time. A great gain about exams like these is that they force you to read upon all – or most – AWS services and therefore you end up with a better overall and holistic view of all the offerings and how they intertwine.

Once again, bound by the agreements and non-disclosures at the beginning of the test I can only disclose a few things.

I used as the main platform to prepare for the test. Exceptional value – I highly recommend it – even if you are an experienced AWS admin, it will provide a great structure. The lessons flow very well and keep you interested and engaged.

In addition to that I would go through all the AWS FAQs and the design documents for all the offerings mentioned in the blueprints. Last but not least, this is not a test you can pass solely by studying. Take advantage of the free tier and roam around as much as possible. The wordpress exercise by is a great one and you should be able to at least complete all those sequences by yourself before going to the exam.

The exam had a lot more situational questions that any practice exam I have seen on the net. You need to really  know your stuff and have practiced quite a lot as the time allotted is just enough for the number of questions given. I was prepared – as I read many people had problem with the time – was reading quite fast all the questions and choices, was answering as quickly as I could and finished with only 5 minutes of extra time to review my flagged ones.

I hope the above will help anyone interested in the exam, AWS is quite marketable right now and offers great value on its cloud offerings, so I would highly recommend both the knowledge and the certification.

Off to study for the professional level, Cheers!


Change IP – Perl Script

It has been a while since my last post – it has been a very busy year both professionally and personally.

A good friend – a rising start in the Linux coding realm – has opened a blog with very interesting posts.

A while ago he created the following script which can be used to seek and replace IP addresses in practically any linux host.

The script has many applications such as migrations or disaster recovery and any similar cases where VMs, or hosts of any nature, need to be restarted at another facility under a different network topology.

The script uses a config file to define which are the files on which it will perform the seek and replace upon. The config file looks like the following:


…allowing you to scan and replace IP addresses on any file practically without altering any other type of content.

The script defines 2 required and 1 optional parameter. the first parameter is the octet value to seek, the second the value to replace it with and the third optional one is the octet position.

For example 11 22 3   - will change IP to

whereas 11 22 - will change IP to

Attaching the script’s files here – I hope they prove to be helpful.


VCAP-DCD passed!

VCAP-DCDsI passed the VCAP-DCD 550 exam this week! I pride myself in my test-taking and approach abilities, but this one really had me worried. With just over 3 minutes left after answering all the questions – obviously there was no time to go back and review any of my flagged answers – I spent a minute or two contemplating what I could have studied better, worrying that once I pressed the finish button, bad news would show up on the screen. Thankfully a passing grade appeared allowing me to leave this one behind and continue towards the VCDX.

There are many great blog posts that helped in my preparation, I suggest you spent a day or two upon searching on them and taking their advice, but as most of them are a bit dated, I will add my two bits just to bring some information up to date.

Nature of the Exam

Like the VCAP-DCA, a really long exam – about 4 hours without any breaks, so make your preparations for it. It had fewer than 50 questions (in some blogs you’ll see as many as 100, so I guess I took an updated version) and there were no multiple choice ones that I can remember. There were a mix of design tool assignments – under 10, with a master design one that you were instructed to allow 30 minutes for, and the rest were multiple drag and drop in multiple containers questions. Don’t let the number fool you, there is barely enough time to complete them.

What makes this exam tough is the nature of the questions, where there is usually a lengthy scenario to read, and the lack of a definite answer (no pick one or two, drag one or two) so you are really not sure if you dragged too many or too few choices to the correct or not containers. My advise, make sure you maintain a certain rhythm, cannot afford to dwell too much on your answers, do your best and move on, cause time will really creep up on you.

Preparation material

Things here aren’t that clear cut either. The breadth of this exam’s questions is quite incredible – the questions will jump from high-level design to technical capacity and capabilities or might even combine them. Here is a list of what I used and think would prepare you sufficiently for the exam:


VCAP5-DCD Official Cert Guide

VMware vSphere 5.x Datacenter Design Cookbook

Video Series

Pluralsight Designing VMware Infrastructure by Scott Lowe

Brown Bag Series on VCAP-DCD


VMware Best Practices and Guides Technical White Papers

VMware Case Studies and Implementation Examples


The material for this exam that you will gather from recommendations, from blogs etc. might seem quite large. This is the basic and major difference between this and other exams. This is not an exam you can cram read for. Allow a few weeks for preparation. Go through the above books twice, go through the videos twice and read as many best practice and guides on the major line of VMware products. Finally, go through at least a dozen architectural case studies and implementation examples. The VMware site has them nicely organized, pick a couple of each kind.

I found that even though this process entailed a lot of reading, it was easy-flow and enjoyable reading. I read/watched all the above twice but didn’t spend any more time on it, neither tried to hard memorize any of it. Instead, I picked a few more VMware design books and booklets and read or skimmed through them once, even standing in a book store. I spent a total of 3 weeks, diligently putting in a couple of hours a day while on a quite hectic work and family schedule, and a lot more time over the weekends.

I am pretty sure I stayed clear of any NDA violations, even though you walk out of this exam pretty “dazed and confused” not being able to remember not even the last questions, let alone the content of the NDA which you agree on at the very beginning.

Also, my “kudos” to VMware for designing a test so well that it can’t be beaten by exam-preparation techniques or question-packs and on the other hand truly manages to test and certify design knowledge and experience, as much on a VMware as well as a general datacenter design level.

I hope I provided a good overall or “holistic”🙂 outlook to the process and exam, that will help anyone interested in taking it. Good Luck!

The power of Custom Visualization

Almost all monitoring/operations tools display a list of critical issues and warnings, display and gather various metrics and plot them in graphs, and most of them have visual dashboards where aggregates are shown depicting totals in various forms – even projecting future warnings. But what most of them lack, is the ability to create custom dashboards hand-picking a bunch of metrics or, even better, allow you to upload your own backgrounds and placing metrics and status(es) on them.

Custom dashboards can be very helpful to operations and service desk teams allowing for a “smarter” and more customizable view of the environment. Lets see a couple of custom dashboards I have created in the past and how helpful they can be:


Here you can see the whole storage fabric within an environment, depicting metrics from fiber switches and various storages. With the ever-increasing abstraction of storage at the virtualization level it is many times easy to overlook what is going on at the bare metal level. Among others here you can see: cpu usage on each node, capacity total/used (also provisioned is denoted where thin disks are used), utilization of each path (verifying multipath, psp etc), queues and response times, iops at the volume and disk levels etc. When the snap was taken a disk had failed in the middle left storage and you can notice the critical status of both the storage and its sp. This dashboard gives a holistic and dynamic view of the storage fabric where behavior patterns can be spotted and investigated even before any warnings would arise.


Here is another one of a simple network layout. It displays statuses on routers, switches, hardware and tunnel interfaces, vrf instances, internet reachability, and speeds of all the connections. (snap was taken late night so speeds are at minimal🙂 ). It can identify where bandwidth is used, help trace issues, identify trace route paths etc.  As network diagrams are very time-consuming to sanitize, I have included the smallest one, but have many of these, at datacenter level, which show many more dependencies and inter-relations.

On larger environments, where there are many different admins, distinct at the storage, network, compute levels, dashboards like these may not be that useful. But on smaller environments, with only a few multi-skilled, multi-responsibility admins and support engineers, dashboards like these help above all bring the environment quickly in mind, show the interconnections and their relations, and finally assist in the quick resolve of any issue and help notice any patterns and future issues.

Diagrams like these have also been created at the application architecture level, but unfortunately there is no way to sanitize those and still keep them meaningful.

Custom design diagrams are an integral part of architectural designs, hand-off documentation and as you can see, operations management.

3PAR cli Commands

Man, I love 3PARs! They are enterpise class storages, packed with features and capabilities, in a uniform line available all the way down to SMB versions. Having fully licensed F-class and 7xxx units we are using for over 3-years now, we can certify as excellent – give kudos to the designers and architects – almost all features and design aspects such as:

– the amazing ASIC processors that handle tasks like Adaptive Optimization, Thin-Reclaiming, tuning volmes to different CPG (raid levels, disk pools), snapshots and so many other tasks flawlessly, quickly and without any disruption of its throughput and performance

– the out-of-band service processor, that monitors and keeps stats on the SAN and facilitates the upgrades in a well designed manner

– the de-coupling of raid groups (GPG) from physical drives allowing every LUN to utilize every single spindle!!!

– the ease of adding shelves or drives, of moving data to different raid-levels or tiers, of expanding LUNS… etc all while everything is hot and without penalty.

– the great full featured IMC admin interface as well as the spartan looking but really complete System Reporter.

3par imc

the all in 1 packed with settings IMC console 


a sample report from system reporter


I could just really go on about these “beasts” of all-sizes that really take a pounding and always manage to achieve much-higher than expected performance, but the reason for this what-started-to-be a wiki style post was to keep record of some great cli commands. (did I mention the great cli command-set?🙂 )

The following command will run every 2 sec – displaying the results in each iteration till you stop it – and give you online stats on your LUNs. Here it is sorted by col 3, which is the avg iops. A lot of great right-this-minute info for monitoring/troubleshooting on the spot, like IOPS, throughput, service time and I/O sizes, queue length.

cli% statvv -sortcol 3 

            r/w I/O per second        KBytes per sec       Svt ms     IOSz KB 
      VVname    Cur  Avg   Max    Cur    Avg     Max   Cur   Avg   Cur   Avg Qlen
...many LUNS
************ t   93  120   241    721   1020    5528  0.21  0.43   7.7   8.5    0
************ t  311  122   364   1846    959    4408  0.43  0.52   5.9   7.9    0
************ t  632  164   857   6692   7170   32110  0.36  0.67  10.6  43.8   10
************ t   45  194   692   2146   8663   57954  3.99  2.04  47.4  44.7    0
************ t  201  278   938  10007  18726   68330  0.37  0.45  49.8  67.3    0
************ t  152  286  1749    854   4063   23486  0.15  0.22   5.6  14.2    0
************ t  151  401  3760   1449  13870  157111  0.34  3.64   9.6  34.6    0
************ t  267  495  3178   3034  19662  179032  0.17  6.15  11.4  39.7    0
************ t  537  539   656   4142   4298    7966  0.20  0.22   7.7   8.0    0
************ t  254  587  3498    939   4576   28225  0.32  0.74   3.7   7.8    0
************ t  104  635  4815   6334  10333   71352  0.90  1.29  61.1  16.3    0
          29 t 2971 4121        39394  95709          0.38  1.66  13.3  23.2   10
Press the enter key to stop...


Same type of command filtered to show ports with current throughput over 10mb (verifying rr operation). All commands have many options and can be filtered to extract exactly what information you need.

cli% statvlun -filt curs,t,bw,10000000
                 r/w I/O per second   KBytes per sec    Svt ms      IOSz KB
Lun name Host  Port  Cur Avg Max    Cur   Avg   Max    Cur   Avg    Cur Avg Qlen
 33 **** ESX6 2:1:1 t 13  17  21  13137 14993 17077  30.67 22.43  987.6 883.3  0
 33 **** ESX6 2:1:2 t 13  17  22  12313 14774 17540  28.74 24.44  953.7 870.4  1
 33 **** ESX6 3:1:1 t 13  17  21  11094 14150 17593  21.67 20.53  858.7 842.3  1
 33 **** ESX6 0:0:1 t 13  17  21  12342 14602 17149  31.00 22.57  955.2 869.2  0
 33 **** ESX6 1:0:1 t 13  17  21  12324 14589 17141  26.32 23.83  953.9 868.4  0
 33 **** ESX6 3:1:2 t 13  17  21  11938 14600 17599  25.02 22.54  924.0 869.1  0
 33 **** ESX6 0:0:2 t 13  17  21  12319 14588 17143  32.67 22.23  953.5 868.3  0
 33 **** ESX6 1:0:2 t 13  17  21  12729 14373 16226  27.21 21.66  985.2 855.6  0
       8           t 104 135     98195 116670        27.92 22.53  946.6 865.9  2


You can also change the sampling periods and the number of iterations. Here used with a stat port command

cli% statport -d 5 -iter 1 -ni

           r/w  I/O per se      KBytes per sec       Svt ms    IOSz KB
  Port  D/C     Cur   Avg  Max    Cur    Avg   Max    Cur   Avg   Cur  Avg Qlen
 0:0:1 Data t  1171  1171 1171  14254  14254 14254   0.74  0.74  12.2 12.2    0
 0:0:2 Data t  1170  1170 1170  14588  14588 14588   0.69  0.69  12.5 12.5    0
 0:2:1 Data t  1503  1503 1503  16036  16036 16036   7.52  7.52  10.7 10.7    0
 0:2:2 Data t  1537  1537 1537  16544  16544 16544   5.20  5.20  10.8 10.8    0
 1:0:1 Data t  1171  1171 1171  14134  14134 14134   0.66  0.66  12.1 12.1    3
 1:0:2 Data t  1173  1173 1173  14277  14277 14277   0.61  0.61  12.2 12.2    1
 1:2:1 Data t  1432  1432 1432  17360  17360 17360   6.03  6.03  12.1 12.1    0
 1:2:2 Data t  1505  1505 1505  18173  18173 18173   4.37  4.37  12.1 12.1    0
 2:1:1 Data t  1171  1171 1171  13752  13752 13752   0.71  0.71  11.7 11.7    0
 2:1:2 Data t  1171  1171 1171  14185  14185 14185   0.67  0.67  12.1 12.1    0
 2:2:1 Data t  1466  1466 1466  28247  28247 28247  24.39 24.39  19.3 19.3   42
 2:2:2 Data t  1492  1492 1492  28899  28899 28899  25.05 25.05  19.4 19.4   35
 2:2:3 Data t  1175  1175 1175  22859  22859 22859  21.53 21.53  19.4 19.4   38
 2:2:4 Data t  1167  1167 1167  22220  22220 22220  21.45 21.45  19.0 19.0   24
 3:1:1 Data t  1170  1170 1170  14513  14513 14513   0.69  0.69  12.4 12.4    2
 3:1:2 Data t  1170  1170 1170  13816  13816 13816   0.64  0.64  11.8 11.8    0
 3:2:1 Data t  1352  1352 1352  24188  24188 24188  23.33 23.33  17.9 17.9    0
 3:2:2 Data t  1370  1370 1370  24335  24335 24335  23.97 23.97  17.8 17.8    0
 3:2:3 Data t  1115  1115 1115  20335  20335 20335  18.32 18.32  18.2 18.2    0
 3:2:4 Data t  1118  1118 1118  20527  20527 20527  18.00 18.00  18.4 18.4    0
    20 Data t 25601 25601      373242 373242        10.52 10.52  14.6 14.6  145


Also, one the newest and most usefull commands, especially for determining if a volume is a good candidate for AO by cheking what percentage of its blocks are heavily accessed

cli% srrgiodensity -cpg *****-CPG -btsecs -1d

------------------Column: acc/(GiB*min) Entry: Acc/min------------------
 CPG_Name  0.0625   0.5     4       32      256    2048   16384   131072
*****-CPG     1.0  17.7 350.1  22100.9  59294.5  6650.8   285.8      0.0


These are the ones that I use more often, there are many more with great options and filters, and they keep on adding more at each version upgrade.