I often find myself with logs in the following format, exported in CSV. timestamp,request_time_in_millis 1545509231863,101 1545509231863,38 1545541592109,202 1545595617945,53 1545595617945,1024 First column is timestamp in milliseconds, second column is timing for some request, also in milliseconds. Often I’d like to see the min, max and average request time, group by day and hour. Here is how […]
How to use Virtualenv in Python to Install Packages Locally
There are two philosophies when it comes to package installation, global first and local first. Global meaning all applications that rely on a certain package have access to the same copy of the library that was installed once. Local means that each project has its own folder of dependencies installed specifically for this project and […]
Crunching Honeypot IP Data with Pandas and Python
I am taking a cyber security class. This week’s assignment had us work on Honeypots. Honeypot is a server that pretends to have a vulnerability of sorts (open ports, old software etc.) and instead collects data on people who are trying to hack it. At the end of the experiment I ended up with some […]
How to Restart Python Script after Exception and Run it Forever
Here is a simple trick that I used to restart my python script after unhandled exception. Let’s say I have this simple script called test.py that I want to run forever. It will just wait 2 seconds and throw an error. import time time.sleep(2) raise Exception(“Oh oh, this script just died”) I use the following […]
Transcribing Speech to Text with Python and Google Cloud Speech API
This tutorial will walk through using Google Cloud Speech API to transcribe a large audio file. All code and sample files can be found in speech-to-text GitHub repo. Transcribe large audio files using Python & our Cloud Speech API. @akras14 shows how https://t.co/dY56lmE0TD — Google Cloud (@googlecloud) January 11, 2018 Sample Results This approach works, […]
How to read CSV file in Python
Reading as a List Let’s say we have the following CSV file, named actors.csv. You can download this file here. The first method demonstrated in Python docs would read the file as follows: Open the file Create a CSV reader Skip first line (header) For every line (row) in the file, do something import csv […]
Convert JSON to dot notation with Python
There are many use cases for converting JSON to a dot notation. For example, my current company used Hue to query our log data. Our logs are stored in JSON, and Hue queries expect a dot notation. For example, this JSON file { “vehicle”: { “car”: { “bmw”: true, “audi”: 1, “ford”: “also good” }, […]
Parsing “Ask HN: Who is hiring?” with Python and Hacker News API
Have you heard of Hacker News? It’s a great mini social network dedicated to all things tech. Once a month they post a thread called “Ask HN: Who is hiring?”, where anyone can list their job openings. With hundreds of comments it quickly gets overwhelming. Turns out it’s very easy to get the same data […]
Simple Python Threading Example
At my work we have a Python library that interfaces to all our API micro services (that are written in Java/Scala). It is a very useful tool for debugging and working with our platform, so I spend a lot of my time in a Python REPL. Often times I find myself needing to hit multiple […]
Fix Issue with Python installed via Brew after OS X El Capitan 10.11.6 Update
Looks like the recent update to OS X El Capitan version 10.11.6 breaks python installed via homebrew. At least it happened to me both on my work and my home machines. If you are experience the same issue, you may see something that looks as following: $ python Python 2.7.10 (default, Oct 23 2015, 19:19:21) […]