Is adopting agile harder for agencies and consulting companies?

Agile development has imprinted solid mark in the software development in last 5-6 years. There is a jump from fringe to the mainstream, and the claim of "We are Agile!" is truer at this time in the teams while it wasn't back in 2010.

The Agile Manifesto was written by a group of software developers for their use. Many of the underlying principles of a framework like Scrum assume the existence of an in-house team and working for the business that employs them.

There is a strong argument for

"Agile Methodology is easier in the settings where you have all the control. Adhering to agile frameworks such as Scrum is more difficult within an agency/consulting structure than it is for an in-house development team especially with the client who understands fixed costs, fixed deadlines, and fixed bid contracts."

On this notion, Atish Narlawar speaks with Christopher Cunningham, a Group Technology Director at Huge, to understands "Does Agile works in the client engagements?" and what are the factors which play a significant role to make it happen.

They start the discussion by going through the factors which have changed in the engagement following Agile process. Chris mentions the importance of a shared language which plays a significant role to create a shift where agencies and client services pushing Agile with their customers. This change has worked with a positive note as agencies started to spend more time delivering excellent products, and less time arguing over out-of-date functional specs. Infact he thinks clients are also requesting and even demanding agile practices from their agencies.

Chris mentions even though some contracts bound with a deadline and fixed costs but there are better ways to get overcome with this, and the best way is to get rid of the fear of uncontrolled cost with the clients. With the combination of informed decisions, tangible results, waste reduction, risk transparency and time to market Agile can certainly give enough ammunition to influence customers and build the trust. With delivering a fully-working product at every milestone, Chris mentions agile expects more involvement from the clients at every step, and corporate companies responded to this need with the positive note.

On the reporting side, Agile driven burn down charts, sprint velocity and points breakdown offers the better way of tracking the progress, identifying the impediments than traditional metrics.

Christopher also notes that within an enterprise environment, Agile methodology is restricted to the software development teams. Other parts of an engagement like business, approval workflows and delivery management still go through traditional way, and that's one of the main reason the clients are not able to leverage the full Agile potential. He predicts there will be a positive change in upcoming days and companies will experience company-wide adoption.

Podcast: Tech Time Podcast

Venue: Brooklyn, NY.

Host: Atish Narlawar

Guest: Christopher Cunningham

React Native Development and Industry Adoption.

React Native is a framework for building native apps using Javascript and React js. Since the announcement which came on early 2015 from Facebook React has been one of the hot topics has ever discussed in Mobile community.

With this notion, Atish Narlawar speaks with Nicholas Brown, Founder of React Native NYC meetup. They start the conversation by jumping to the React JS component based framework and then natural progression of extending the web framework to mobile platforms through React Native.

Nicholas briefs the Architecture and introduces the feature of converting Javascript functionality into native platform code. Nicholas clarifies how React Native differs entirely from existing Javascript based mobile frameworks like Cordova and Appcelerator Titanium. He mentions within a year, the number of contribution to React Native exceeds that to the Cordova.

He explains the performance impact of using React Native compared to directly developing with Native code, especially during the transition of Javascript to Native code. However, to address this concern, better features and tools has been evolved like live reload and Microsoft CodePush. He also goes through various use cases where developing a mobile solution using React Native turns to be better as compared to native ios/android code.

Nicholas thinks once the solution gets developed on one platform, for example, iOS then the same solution can be used for Android with the minimal efforts around of 10% of the total. The discussion then goes through the current adoption of React Native in the tech industry and how most of the company are gearing up themselves meanwhile taking the cautious approach to be ready for next real change. He also shares the Google's attempt to enter into this domain with Flutter.

The discussion winds up with the recommendations for the beginners, with Facebook own tutorial and plugin repository named as React Parts for Native and Web plugins.

Podcast: Tech Time Podcast

Venue: Brooklyn, NY.

Host: Atish Narlawar

Guest: Nicholas Brown

Meetup: React Native NYC

Links: Facebook React, React Parts, Flutter

Search Engine Optimization in 2015.

Search engine optimization (SEO) is the process of affecting the visibility of a website in a search engine's especially unpaid results. In general, the earlier and more frequently a site appears in the search results list, the more visitors it will receive from the search engine's users.

As the internet evolves over the time, it also showed impressive progression about the changes in user behavior about how, why and what they search for. This creates a continuous battle of how/why search engines prefer to represent information and creates a wide array of variations for SEO team to focus on.

One of the key impression with SEO, it is ever changing landscape. With this notion Atish Narlawar talks with David Sosnowski, a technologist, Group Director of Marketing at Huge.

They start the discussion by going through the ever-changing landscape of SEO. David explains even through SEO are changing fast to adapt the users and SEO analysts also needs to evolve over the period but the fundamental SEO principles have remained same even in 2015. He explains the core fundamentals user signals like bounce rate, ATOS (average time on site), ATOP (average time on page), conversion rate, click throughs which still be effective to guide SEO strategies.

With stressing Links and URLS as the most important factor, the discussion outlines few simpler and effective steps for any SEO strategy,

  • Understand the areas of improving indexation of the site through using Google Webmaster tools.
  • Task the SEO and technology teams to enhance coding elements.
  • Work with content teams to make sure expected voice matches how audience is searching through keyword research.
  • Monitor links coming into the site with tools like h-refs and teach your content, communication and social teams on the importance of links.

To deal with high volume CMS based website (For eg. a website with 70,000 pages) he recommends to focus first on categorization and second to rely heavily on google analytics based on understanding what is happening not just reporting numbers.

David shares the insights about how to keep and stay ahead in SEO Learning curve. Heclears the misconception revolves around redirects 401, add notes to the security and keyword selections. David recognizes the importance of Machine learning through the introduction of Google ‘RankBrain’ and explains SEO is turning to be a team work with co-ordination with other disciplines like UX and Products.

The discussion concludes with the mantra that SEO is no different, and definitely, it's not a rocket science. He recommends coming up with the roadmap of “Do, Report, Understand, Do.”

Venue: Brooklyn, NY.

Host: Atish Narlawar

Guest: David Sosnowski

Twitter: @SEO_Raptor

BEM - CSS Methodology.

Since the age of Geocities Era, CSS has been evolved a lot. It has seen the transition from plain CSS to emerging development framework such as such as OOCSS (object oriented CSS), SMACSS and now BEM.

BEM – stands for block, element, modifier. By definition, it is a CSS development methodology specifically designed with flexibility and ease of modification in the mind. In the given podcast episode, Atish Narlawar talks to William Anderson about the BEM. William is a die hard Technologist, Software Engineer and part time Professor at The New School University New York.

The conversation gets a start by going through high-level CSS evolution since the the 1990s. William shares the fascinating things in the CSS in 2015 like logical integration, media queries, and animations. Along with he also explains how CSS development became quite complex down the line due to cascade, responsive design and the javascript itself.

One of the major challenges gets added with the new found complexity includes messy CMS, broken semantics and SEO refactor and BEM emerged as one of the possible solution to address these challenges.

With the example of feature card, William explains how BEM tries to fix the issues mentioned, such as complexity, semantics, and decoupling. Using zero specificity, he shares how seamlessly his team were able to integrate SEO tag change in the live project.

William also shares his valuable insight in terms of how BEM works with other CSS methodologies such as OOCSS (object oriented CSS) or SMACSS (smacks). He tries to clarify the long haunting doubts about BEM such as BEM is too verbose and it pollutes the DOM with Blocks, Elements, and Modifiers all cooked in class names. He shares what could be the best strategy for initiating the adoption of BEM into existing projects, and quite possible barriers team may face during the transitionary switch.

At the end, he also shares valuable resources where someone should start digging about BEM methodology and its related practices.

Venue: Brooklyn, NY.

Host: Atish Narlawar

Guest: William Anderson

Twitter: @TheWAAnderson

Boot the Swagger: API as good as it's documentation.


Swagger provides one of the best representation of the REST API during development. It provides interactive API documentation baked in with the application itself. Contrary to create separate specification document with cost of maintenance and risk of obsolete, Swagger generates the API documentation on the run time with updates. It also provides user friendly REST interface to test the code at the same place. This idea of live documentation has real value, and I guess you should give a try during API development.

In one of the recent engagement, I had to showcase the working proto of Swagger to the client, and being said easy to setup I estimated it for 15 mins. But I was damn wrong, I had to spend 2 hours to get Swagger up and running. So, I had an idea why not to just simplify the Swagger configuration process in very simple steps, so hungry user like me can configure in existing application and see the benefits.

I configured it for Spring Boot, but I fairly assume it should be similar in most Swagger supported frameworks.

Add dependency.

In pom.xml

Add Configuration Class

Add annotations.

For every REST endpoint methods of Controller, @ApiOperation annotation needs to be added. For eg,

Add Swagger UI HTML library from GitHub.

Download Swagger UI Library v2.1.1 from Github into resource/public folder. Replace the declaration of Url from the first index.html function,

These are minimal steps required to spinup swagger on existing Spring Boot web application. Swagger API configuration and Swagger UI documentation can be seen at

http://localhost:8080/api-docs

http://localhost:8080/index.html

Note

Swagger has been updated with Specification 2.0. Version used in the example may go obsolete in near by future.

CDN for API Orchestration.


Content Delivery Network(CDN) is a globally deployed distributed network of servers that serves the content with high accelerated, high performance, and high availability. Traditionally CDN is being used mostly to serve static assets like images, videos, JS/CSS. With the recent trend of API based service developments, the adoption of CDN is also getting involved with serving API responses.

API content delivery network (API CDN) is a version of CDN that specifically focuses on API caching and other benefits. They do not maintain any business logic, but only serve copies of content for requests they have seen before and have previously pulled from the original source. Every request offloaded to an API CDN is one less request to the application servers. Serving requests from an API CDN are usually cheaper in terms of bandwidth cost than serving same bandwidth from servers. API CDN can also reduce latency by caching content in data centers across the country or world and deliver responses from the closest available node to users.

While API CDN is similar to a regular CDN, the key differentiator of an API CDN is that it provides critical API-specific features (see below) in addition to static content delivery.

API CDN Features.

  • SSL support. You can own SSL certificate.
  • Geographical distribution.
  • Token authentication and Management.
  • DDoS mitigation. An API CDN should be able to recover from attacks without significant service degradation.
  • Web Application Firewall. An API CDN provides support for rules automatically mitigating XSS or injections attacks.
  • Support for SPDY and/or HTTP/2.
  • API for URL invalidation and other cache management functionality.
  • SLAs and support process.
  • Effortless up and running.
  • Support to Fixed and per request/bandwidth pricings.

There are plenty of commercial players in CDN market, but there are few specialized in serving API over CDN.

Fastly

Fastly specializes in API caching. It serves a significant number of large-scale tech-savvy clients including Github, Twitter, Yelp, Etsy and more. Fastly supports. It has supports to SSL, Geo, Token Authentication and easy maintenance. From the pricing perspective, it seems to be very competitive and can be online instantly with a trial account. It also supports SAN certificates.

The only thing missing from Fastly support to a Web application firewall.

Mashery/Intel

One of the original players in the API CDN space. Very mature product with a large number of enterprise clients. It supports most of the features listed above including a Web application firewall. One of the not so good thing about Mashery compared to Fastly is unclear pricing structure and slow setup.

Akamai

The biggest CDN vendor in the world. Akamai is known primarily for hosting static web traffic and does not have a dedicated API solution. While it may be possible to make Akamai work as an API CDN, but doing a deep analysis of Akamai for this evaluation due to Akamai’s limited functionality in this area is time consuming as compared to the other platforms under consideration.

DIY Varnish

The self-hosted option. Varnish is the most popular open source cache server. It would be possible to deploy Varnish instances across availability zones. This option will likely cost a significant amount in terms of additional DevOps hours and hosting and will not be as full-featured as fully-managed products.

Cost Estimation.

Consider the site has 500,000,000 requests 15,000 GB of bandwidth requirement for a certain time frame. Pricing is generally based on usage, we can estimate the price on the based average usage.

Based on estimated traffic, the Fastly cost of about $1,975/Month(Price is based on $0.0075 per 10,000 requests and $0.12 per GB and $100/mo for certificate maintenance). Mashery price calculation is available over the phone but with previous records it would easily cost more than $5,000/Month.

Using the AWS pricing calculator, we can estimate the cost of usage and bandwidth for an uncached site through application servers. Consider we have a need to serve 15,000 GB of outbound traffic for a month, we would need 10 large servers (in terms of CPU, RAM, etc). The cost of this would be around $2,500/mo to Amazon. If we are able to cache most API requests, we can reduce usage to 4 medium servers and 2,000 GB at a monthly cost of $750, leading to a cost reduction of $1,750.

This infrastructure saving of $1,750 can be largely offset by the cost of Fastly API CDN while getting away additional number of critical features at the same time.

Summary

API CDN has a list of significant features including speed, DDoS support, token authentication at a very minimal price. Most of the cost can be covered by saving bandwidth on servers. There are a couple of players in API CDN with proven track record with large enterprise clients, upfront about capabilities, service levels, pricing, and have excellent documentation.

Docker Cache Insights.


I have gained various insights over the time about the better way of using the feature of Docker Caching for the couple of scenarios. I have tried to detail the couple of them which might be helpful for the engineering teams to be more productive and efficient.

About the Docker Cache, Docker provides the functionality to package an application in such a way that “Build once and run anywhere”. During development and continuous deployment, someone has to deal with dozens of continuous docker builds. To facilitate faster turnarounds, docker has the ability to cache the various steps of the build process so that subsequent builds are almost instantaneous. Yes, we all know that Docker is inbuilt with the caching and here are some details about the insights.

Dealing with source control operations.

To get a copy of source code in docker container, developers prefers to use git commands in or wrap into the a script and use Docker’s ‘RUN’ step.

# Clone the git repo
RUN /bin/sh /git clone git@github.com/<account>/<repo>.git
# Git repo operations in a script
RUN /bin/sh /git-clone.sh

Docker clones the repository for that first time and then caches the RUN command executes instantaneously for subsequent builds. Great! but it causes an issue, if repository gets updated after cache and docker build new container, it uses previously cached version as an output resulting a stale codebase. The expectation here is an output of an RUN command should be updated each time of new container build. As it stands, unless the RUN command itself changes (and thus invalidates Docker’s on-host cache), Docker will reuse the previous results from the cache. In this case once ‘RUN git clone*’ gets executed once, the subsequent build uses the same copy of the code and won't try to execute each time to get latest changes.

One quick way to get rid of this is disabling the cache by building with ‘–no-cache’ flag. But it invalidate all the build steps and not a specific step of ‘RUN command’ resulting in the execution of all steps again and totally defeating the purpose of the cache. One of the way to deal with this is to generate unique RUN command each time ensuring it gets run each time

  1. We can wrap the Docker build in another script that generates a uniquely numbered mini script for the clone operation. This step would insert the invocation of that script into the Dockerfile that is generated on-the-fly just prior to build time, such that for the operation that must be run every time – the clone – its RUN statement is indeed unique.

    i.e. RUN /bin/sh /git-clone-123abc.sh

    where ‘-123abc’ is uniquely generated and appended for each build (and subsequent executions create something like ‘clone-124abc.sh’) and contains the git clone operation.

  2. Place this source control operations into the last RUN that is listed in the Dockerfile. This guarantees that Docker will run the clone during each build while having the advantages of being both fully automated and ensuring that the cache is used right up to that last unique RUN.

Appropriate usage of ‘ADD’ step in Dockerfile.

‘ADD’ step is being used most commonly in Dockerfile. But I think if it’s not been used properly, it becomes a common cause of cache busting during builds.

Let me give an example through Node.js MEAN stack docker build steps,

# 1. Add the application folder. It has package.json
        ADD . /src
# 2. Go to the package.json directory and build the app using npm package manager.
        RUN cd /src && npm install

The first step adds the current folder to container’s /src folder. The second step is very time-consuming as it installs dependencies. But at the same time package.json don’t get change often, hence during subsequent build this step should be very fast.

But here is the glitch. In the first step, when Docker does its comparison either to use cache, it compares the folder (the ‘.’) against the previously built folder, and if any file has changed from the current folder in meantime, the cache gets busted and second step of ‘npm install’ get executed even though package.json didn’t change.

This behavior has been ignored many times. This can be avoided with reordering of the steps.

# 1. Add package.json
    ADD package.json /src/package.json
#2. Run the Build with package.json
    RUN cd /src && npm install
# 3. Add the application folder.
    ADD . /src

In this case, even though any file in application folder gets change, it doesn't affect the first two steps unless package.json itself change. Docker caches the first two steps easily and gets bust third command onwards.

mTime(modified time) TimeStamp

Docker determines whether or not to use the cached version of a file is by comparing several attributes of the older and the newer version, including mtime. For example, during the MEAN stack build steps

# 1. Add package.json
    ADD package.json /src/package.json

After the first docker build, docker uses the cached copy of ‘package.json’ during subsequent builds if docker build is happening on the same project from the same directory several times in a row.

But most of the time, engineering team uses continuous delivery environments like Jenkins or CD. So to configure docker hook up with the build trigger a fresh clone of the application’s repository occurs. When this happens the mtime of a new copy of ‘package.json’ gets different than the previous cached ‘package.json’. And this is a problem for docker. As Docker uses the mtime of a file when doing its comparison and the mtime changes for our package.json file on every single clone, the cached version can never be used although technically file content never changed.

Fortunately, there is a solution! As part of our build process, after cloning the Git repository mtime of the ‘package.json’ The file needs to be changed to the time file was last changed within Git. This means that on subsequent clones if the file has not been modified according to Git, then the mtime will be consistent and thus Docker can use the cached version.Below sample script is one of the way to deal with mTime Timestamp.

#!/bin/bash
# 1. Get the git revision for package.json
    REV=$(git rev-list -n 1 HEAD 'package.json');
# 2. Get the timestamp of the last commit
    STAMP=$(git show --pretty=format:%ai --abbrev-commit "$REV" | head -n 1);
# 3. Update the file with the last commit timestamp
    touch -d "$STAMP" package.json;

Step 3 ensures mTime is always same for all the git clone until unless the updated version of the package.json file gets checked into source control.

This solution is just a workaround. There are a couple of proposed enhancements to the behavior of docker caching strategy with consideration of other attributes like sha1sum and md5sum along with a timestamp.https://github.com/docker/docker/issues/4351.If this feature gets added then workaround mention here not be needed.

Avoid installing unnecessary packages

Every docker file starts with system installation instructions like,...

RUN apt-get update && apt-get install wget -yy
RUN wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | apt-key add -
RUN echo "deb http://pkg.jenkins-ci.org/debian binary/" >> /etc/apt/sources.list
RUN apt-get update && apt-get install -y -qq --no-install-recommends git jenkins curl telnet unzip openssh-client && apt-get clean

Throughout development, more and more package gets added. Same time developers generally don’t try to refactor this portion of dockerfile considering it is a very core part of the docker build and package removal might encounter unnecessary issues down the line. As per my experience working with the a big team, this section becomes the hub of unnecessary packages.

The solution is simple, just have continuous refactoring of setup steps and avoid installation of unnecessary packages. Breaking the lengthier steps into multi-line and then sorting the arguments also help to get rid of duplicates.

RUN apt-get update && apt-get install -y -qq --no-install-recommends\
git \
jenkins \
curl \
telnet \
unzip \
openssh-client \
&& apt-get clean