Archives for October 2016

Git grep multiple repos at once

October 25, 2016 by Alex Kras 2 Comments

Share this:

My company uses multiple git repos that all depend on one another. Often I find myself trying to grep through a few of them at the same time.

Regular grep works, but it takes a long time and displays a lot of noise. git grep only searches through files that are NOT listed in .gitignore file, but only works at per-repo level.

Few months ago I wrote a script to walk through my repos and generate a list of things I worked on that week.

I’ve modified that script to do git grep for all git repos.

i.e. git-grep-all "some string" will produce:

repo-1
/some/file: some string
/some/other/file: some string 123

repo-2
/yet/another/file: again some string

repo-1

/some/file: some string

/some/other/file: some string 123

repo-2

/yet/another/file: again some string

I made a folder with some popular JavaScript repos, and running git-grep-all TODO will produce the following result.

grep-demo$ git-grep-all TODO

preact
/Users/akras/Desktop/grep-demo/preact/config/codemod-let-name.js:8:     // @TODO unsafe, but without it we gain 20b gzipped: https://www.diffchecker.com/bVrOJWTO
/Users/akras/Desktop/grep-demo/preact/test/browser/lifecycle.js:504:                                    // @TODO Component mounted into elements (non-components)
/Users/akras/Desktop/grep-demo/preact/test/browser/spec.js:97:          // @TODO: migrate this to preact-compat

rendertron
/Users/akras/Desktop/grep-demo/rendertron/src/cache.js:29:    // TODO(samli): check info (data[1]) and loop through pages of entities to delete.
/Users/akras/Desktop/grep-demo/rendertron/src/cache.js:107:// TODO(samli): Allow for caching options, like freshness options.
/Users/akras/Desktop/grep-demo/rendertron/src/renderer.js:56:      // TODO(samli): This needs to change to use the non-deprecated API after Chrome 61

vue
/Users/akras/Desktop/grep-demo/vue/.gitignore:5:TODOs.md
/Users/akras/Desktop/grep-demo/vue/benchmarks/dbmon/lib/memory-stats.js:61:     // TODO, add a sanity check to see if values are bucketed.
/Users/akras/Desktop/grep-demo/vue/packages/weex-vue-framework/factory.js:6383:        // TODO: record before patch positions
/Users/akras/Desktop/grep-demo/vue/packages/weex-vue-framework/factory.js:6417:    // TODO: finish implementing move animations once
/Users/akras/Desktop/grep-demo/vue/src/platforms/weex/runtime/components/transition-group.js:63:        // TODO: record before patch positions
/Users/akras/Desktop/grep-demo/vue/src/platforms/weex/runtime/components/transition-group.js:97:    // TODO: finish implementing move animations once

weweChat
/Users/akras/Desktop/grep-demo/weweChat/README.MD:45:### TODO

grep-demo$ git-grep-all TODO

preact

/Users/akras/Desktop/grep-demo/preact/config/codemod-let-name.js:8: // @TODO unsafe, but without it we gain 20b gzipped: https://www.diffchecker.com/bVrOJWTO

/Users/akras/Desktop/grep-demo/preact/test/browser/lifecycle.js:504: // @TODO Component mounted into elements (non-components)

/Users/akras/Desktop/grep-demo/preact/test/browser/spec.js:97: // @TODO: migrate this to preact-compat

rendertron

/Users/akras/Desktop/grep-demo/rendertron/src/cache.js:29: // TODO(samli): check info (data[1]) and loop through pages of entities to delete.

/Users/akras/Desktop/grep-demo/rendertron/src/cache.js:107:// TODO(samli): Allow for caching options, like freshness options.

/Users/akras/Desktop/grep-demo/rendertron/src/renderer.js:56: // TODO(samli): This needs to change to use the non-deprecated API after Chrome 61

vue

/Users/akras/Desktop/grep-demo/vue/.gitignore:5:TODOs.md

/Users/akras/Desktop/grep-demo/vue/benchmarks/dbmon/lib/memory-stats.js:61: // TODO, add a sanity check to see if values are bucketed.

/Users/akras/Desktop/grep-demo/vue/packages/weex-vue-framework/factory.js:6383: // TODO: record before patch positions

/Users/akras/Desktop/grep-demo/vue/packages/weex-vue-framework/factory.js:6417: // TODO: finish implementing move animations once

/Users/akras/Desktop/grep-demo/vue/src/platforms/weex/runtime/components/transition-group.js:63: // TODO: record before patch positions

/Users/akras/Desktop/grep-demo/vue/src/platforms/weex/runtime/components/transition-group.js:97: // TODO: finish implementing move animations once

weweChat

/Users/akras/Desktop/grep-demo/weweChat/README.MD:45:### TODO

Few notes to get it working

You need to update the BASE variable. All my git repo are nested under one folder, and it looks as follows:
- company-name
  - repo-1
  - repo-2
  - repo-3
I’ve saved this file to ~/bin/git-grep-all, made it executable, and added ~/bin to my PATH.
Now I can just run git-grep-all "my search string" to search through all repos that I have cloned.

#!/usr/bin/env python

import commands
import os
import sys

BASE = 'path/to/multiple/repos/to/search/' # <----------- Fix base path

if len(sys.argv) < 2:
    print 'used as: grep-all "some string"'
    exit(1)

search = sys.argv[1]

repos = os.listdir(BASE)
repos = [os.path.join(BASE, repo) for repo in repos]


results = ""
for dir in repos:
   command = 'cd ' + dir + ' && git grep --color="always" \'' + search + '\''
   results = commands.getstatusoutput(command)[1]
   if(
       len(results) > 0 and
       "Not a directory" not in results and
       "Not a git repository" not in results
   ):
       print "\n" + os.path.relpath(dir, BASE)
       print results

#!/usr/bin/env python

import commands

import os

import sys

BASE = 'path/to/multiple/repos/to/search/' # <----------- Fix base path

if len(sys.argv) < 2:

print 'used as: grep-all "some string"'

exit(1)

search = sys.argv[1]

repos = os.listdir(BASE)

repos = [os.path.join(BASE, repo) for repo in repos]

results = ""

for dir in repos:

command = 'cd ' + dir + ' && git grep --color="always" \'' + search + '\''

results = commands.getstatusoutput(command)[1]

if(

len(results) > 0 and

"Not a directory" not in results and

"Not a git repository" not in results

print "\n" + os.path.relpath(dir, BASE)

print results

Note: This method is not perfect. It will only grep repos at the state that you last left them in and it will not pull new changes from origin. Still, I find that it works really well for 80% of my use cases.

Hope it helps.

Please Share:

Wanna see my pretty face and hear me ramble? Check out my YouTube channel 🙂

P.S. Check out Disrupted: My Misadventure in the Start-Up Bubble by Dan Lyons a co-producer and writer for the HBO series Silicon Valley (sponsored).

P.P.S. Let's connect on Twitter, Facebook or LinkedIn. You can also get updates via RSS Feed or the Email Form Bellow.

Follow @akras14

Subscribe to Blog via Email

Share this:

Google May Be Stealing Your Mobile Traffic

October 15, 2016 by Alex Kras 255 Comments

Share this:

Update 10/17/2016 6:30pm: You can read a response from the tech lead of the AMP Project for Google, in the comment area.

Update 11/01/2016 10:40pm: After learning more about AMP, it became clear that the original title was inaccurate. A better title would have been You may be losing Your Mobile Traffic.

Update 11/03/2016 10:30pm: I was invited to have lunch with Google AMP team, where I got to discuss some of my concerns. I’ve summarized my key takeaways here.

Update 01/14/2017 10:00am: Navigation Toolbar is now also scrollable on Mobile Safari.

Update 02/08/2017 10:00am: AMP Toolbar Now Has A Button To View and Copy the Original URL.

Update 09/19/2017: While I still have concerns with AMP, I wanted to point out that it did make things better for users and there is currently no good alternative.

I run my blog on WordPress with a custom theme. 6 month ago I’ve added support for Accelerate Mobile Pages (AMP) in order to provide better user
experience for mobile devices. I did not research the AMP project well and have made some costly false assumptions that I wanted to share.

Most importantly, I was surprised to find out that instead of redirecting users to an optimized version hosted on my server, Google was actually serving a snapshot of the page from their own cache. To make things worse, Google was injecting a large toolbar at the top of the snapshot encouraging users to get back to Google search results (a functionality already provided by the back button) and making it harder to get to the original site.

@akras14 @rauchg @cramforce @Errol_Flynn_ @benthompson I too thought AMP was fast HTML subset (static check), missed central service connex.

— BrendanEich (@BrendanEich) November 12, 2016

To draw a contrast, Google have been keeping snapshots of sites for a very long time, available via a “cache” link right next to search results. The cache link, however, was optional. Clicking on the search result, traditionally, would take the user to the original source. With AMP, Google have turned this equation around.

I do believe that Accelerated Mobile Pages is a great project! At the same time, I find the way Google handles redirects slightly dishonest and very concerning.

Accelerated Mobile Pages

Accelerated Mobile Pages (AMP) is a newish open source project that Google launched in the February of 2016.

In their own words:

AMP is a way to build web pages for static content that render fast. AMP in action consists of three different parts:

AMP HTML is HTML with some restrictions for reliable performance and some extensions for building rich content beyond basic HTML. The AMP JS library ensures the fast rendering of AMP HTML pages. The Google AMP Cache can be used to serve cached AMP HTML pages.

You can watched this promotional video to learn more about the project.

Most Web Developer would recognize the suggested list of features (listed below) as a great thing.

Allow only asynchronous scripts
Size all resources statically
Keep all third-party JavaScript out of the critical path
All CSS must be inline and size-bound
Font triggering must be efficient
Minimize style recalculations
Only run GPU-accelerated animations
Prioritize resource loading
Load pages in an instant

As an added bonus, it was rumored that Google will prioritize AMP optimized pages in their search results, so there was really no downside in giving it a try. Or so I thought.

AMP In Action

I run my blog on WordPress, so it was really easy to add AMP support via the AMP plugin. I remember bragging that it took less than 5 minutes to set it up, saying “Google wants, Google gets”.

I’ve been running my site with AMP support for over 6 months now.

If you head over to Google and search for “Git Tips” you will see my site with the APM icon next to it.

1-amp-demo

Clicking on that link will open up AMP friendly view of the post.

The Problem

So far everything seems great, with one big problem. The view above is not being served from my site, but rather from Google Cache.

The URL of the page looks as follows:

https://www.google.com/amp/s/www.alexkras.com/19-git-tips-for-everyday-use/amp/?client=safari

Essentially, the original link with www.google.com/amp/s/ added in front of it.

Which is actually a good thing. It reduces load on my site and allows readers to access to the information much faster that my $10 a month server could accommodate.

So Why in the world am I complaining, even going as far as to imply that Google might be stealing my mobile traffic?

Guess what happens when the “close” button is clicked inside the AMP view?

I was expecting it to cause a redirect to the original article. Instead it redirects back to Google search results. Say What? How are people supposed to get to my site?

Luckily in my case there is a blue header at the top, clicking on which would redirect to the homepage. I have seen a number of sites, however, that do not have such header. In those cases, I literally had to copy the URL and paste it into my browser to get to the original site. I am afraid that most readers are not willing to go through that trouble.

Also note that even with the header in place, landing on the home page will make it really hard to locate the original article returned via the Google search.

One final nitpick. If Google cares so much about the mobile experience, why cover 15% of the small mobile screen with a fat bar at the top?

What about the little guy or gal?

If you watched the AMP promotional video, you’ve heard them talk about the web as the greatest innovation of our time. Except the user experience was broken on mobile and AMP set out to fix just that.

I think Richard Gingras, Head of News at Google, said it best and I quote: “Let’s make the web great again“.

Note: To be fair, Richard Gingras said it way before Donald Trump made it his campaign slogan. Still I couldn’t resist to point that coincidence out.

great-again

Unfortunately, in the process of fixing the Web, Google broke something else. It used to work like this:

Search Google
Find interesting result
Go to the site
Explore the site further OR hit the back button to go back to Google search results

Now it works like this:

Search Google
Find interesting result
Read the content without leaving Google
Try to explore the original site AND get redirected back to Google search results.

The web used to be the place where anybody could publish quality content, help thousands of people, and earn a few dollars along the way. Google Adsense was one of the main products that made this reality possible.

A lot of authors hope that people would find their content useful, stay for a while and come back in the future. None of this will work if readers are not able to get to the site in the first place.

In addition, given the wide spread of Ad Blocking software, mobile platform remains one of few places where ad revenue continues to be a viable option.

By hijacking the mobile traffic and keeping users from leaving their site, Google gets to benefit from somebody else’s content while at the same time displaying their ads. This cuts further into already narrow margins of independent publishers.

What can be done?

Ask Google to give users an easy option to view the original post.

Google could change the close button to take users to the original site. Alternatively, a different button can be added to provide users with such option.

Make sure your AMP content is optimized for the new deal

I think it is important to point out that Google is playing by the AMP rules. My main concern is that a lot of people (like myself a week ago) have yet to realize that AMP search results are being treated differently from other search results.

In other words, Google is playing by the rules, but the rules have changed.

When I installed the AMP plugin, I assumed that all my setting from my main site will carry over. This was not the case.

The AMP plugin extracted the text content and stripped everything else out. I noticed some of it, but did not, for example, realized that it removed my Analytics code. I wasn’t too concerned with the changes that I did notice, because I assumed that users will have an easy way to drop back to my original site. As of writing of this article, there is no such option.

That being said, Google allows AMP sites to display whatever content they like, as long as it complies with AMP rules.

I recommend each AMP enabled side to do the following

Make sure your Analytics tracking code is properly installed
Make sure you have a header that points back to your main site
If your site has a menu, make sure it is visible in the AMP version
Consider adding a link at the top of your AMP page, giving user an option to visit the original post
Make sure your ads and other promotional material is properly integrate into the AMP version

Conclusion

I hope I am not being too hard on Google. I get it. Real innovation is hard. It takes courage. No one can get everything right on the first try and it’s easy to overlook some details. Since AMP is an open source project, I assume my feedback is welcomed.

That being said, it’s the people, millions of authors who take their time to write and billions of readers who take their time to read, comment and share that make the Web great.

Google is improving things for the mobile readers with the AMP project. I only wish they could do so without hurting the content creators.

P.S. This post has generated an interesting discussion on Hacker News, with some good examples both supporting and opposing my arguments.

Please Share:

Wanna see my pretty face and hear me ramble? Check out my YouTube channel 🙂

P.S. Check out Disrupted: My Misadventure in the Start-Up Bubble by Dan Lyons a co-producer and writer for the HBO series Silicon Valley (sponsored).

P.P.S. Let's connect on Twitter, Facebook or LinkedIn. You can also get updates via RSS Feed or the Email Form Bellow.

Follow @akras14

Subscribe to Blog via Email

Share this:

Parse JSON Object to Map in Scala Using Jackson

October 6, 2016 by Alex Kras 1 Comment

Share this:

I couldn’t find a good example online, so here it goes.

import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.databind.ObjectMapper

val data = """
    {"some": "json data"}
"""

val mapper = new ObjectMapper

mapper.registerModule(DefaultScalaModule)
mapper.readValue(data, classOf[Map[String, String]])

import com.fasterxml.jackson.module.scala.DefaultScalaModule

import com.fasterxml.jackson.databind.ObjectMapper

val data = """

{"some": "json data"}

"""

val mapper = new ObjectMapper

mapper.registerModule(DefaultScalaModule)

mapper.readValue(data, classOf[Map[String, String]])

Hope it helps

Please Share:

Wanna see my pretty face and hear me ramble? Check out my YouTube channel 🙂

P.S. Check out Disrupted: My Misadventure in the Start-Up Bubble by Dan Lyons a co-producer and writer for the HBO series Silicon Valley (sponsored).

P.P.S. Let's connect on Twitter, Facebook or LinkedIn. You can also get updates via RSS Feed or the Email Form Bellow.

Follow @akras14

Subscribe to Blog via Email

Share this: