Connecting to Heroku Postgres in Python

This is for Python 3, using psycopg2:

import psycopg2
import subprocess

proc = subprocess.Popen('heroku config:get DATABASE_URL -a my-heroku-app', stdout=subprocess.PIPE, shell=True)
db_url ='utf-8').strip() + '?sslmode=require'

conn = psycopg2.connect(db_url)

Make sure you’ve installed Postgres locally with SSL support. Here’s how I did it:

brew install postgres --with-openssl
pip3 install psycopg2

For some reason, did not work for me. Instead, I needed to use the Homebrew version.

Using Pandas to make tables in Jupyter

If you are using Jupyter, you can use Pandas to make nice tables from SQL queries:

from sqlalchemy import create_engine
import pandas as pd
import subprocess

proc = subprocess.Popen('heroku config:get DATABASE_URL -a my-heroku-app', stdout=subprocess.PIPE, shell=True)
db_url ='utf-8').strip() + '?sslmode=require'

engine = create_engine(db_url)
pd.read_sql_query('select id, email from users', con=engine)

That will give you a nice table like this:

Memory Issues with NodeJS Streams? Remove 0.8.x Streams from Your Pipe

I was working on an ETL process with NodeJS that ran on Heroku, which limits you to 500MB of memory on a standard dyno. The ETL was quickly running out of memory, even through each document that it processed was less than 1MB.

This ETL was made up of a Readable stream from a MongoDB Cursor, then two Transform streams, one using event-stream and one implemented as a subclass of Stream.Transform. The last stream was a Writable stream that validated and wrote back documents to MongoDB.

Clearly the readable stream was reading documents out of Mongo way too quickly for the transform streams to do their work and the writable stream to write the documents back to MongoDB. But wasn’t NodeJS suppose to manage this automatically, assuming the highWaterMark was set to something reasonable?

It turns our that, yes, NodeJS does manage this correctly, as long as every stream in your pipe is a 0.10.x stream or greater. NodeJS implemented the highWaterMark in Node 0.10.x.

The culprit in my case was event-stream, which is an old library that only makes 0.8.x streams.

Moral of the story: if you are having memory issues with streams that seem like backpressure should be solving, make sure all of the streams in your pipe are 0.10.x streams or greater!

Learning React as an AngularJS Developer

React isn’t a framework

You use Angular, so you probably like frameworks. React and redux are not frameworks in the same sense as Angular. You can use React Redux Starter Kit and/or Redux CLI to get a React/Redux app set up with some sensible defaults, though.

Replacements for other parts of Angular

  • $httpaxios
  • Factories and services – ES6 modules with Webpack
  • Mocking modules (or in Angular, services or factories) in your tests – babel-plugin-rewire

Understand the component lifecycle methods

Best resource I’ve found is Understanding the React Component Lifecycle

Get a good authentication library

I recommend redux-auth-wrapper.

A Better $patch Method for Angular's ngResource

Angular’s ngResource makes a pretty decent starting point for developing a front-end model layer, but tends to be a little simplistic as your backend API gets more sophisticated.

One issue I ran into recently is that the $patch method sends the entire object in the request body. This is usually not what you want to do, because the PATCH method is designed for a partial resource modification, typically using something like JSON Merge Patch or JSON Patch as the request body.

Here’s how to add a $patchFields method to your resource that creates a JSON Merge Patch for a set of fields, and then sends it to the backend.

This is how you’d use it:

var product = new Product({
   id: 231
   name: 'iPad',
   price: 429.99,
   size: {
      height: 9.4,
      width: 6.2,
      depth: 0.2

product.price = 400;
product.size.height = 10;

var promise = product.$patchFields(['price', 'size.height']);

 * Sends a request like this:
 * {
 *    "price": 400,
 *    "size": {
 *       "height": 10
 *    }
 * }

And here’s the code:

angular.module('myApp').factory('Product', function($resource) {

   var Product = $resource(
      { id: 'id' }

   angular.extend(Product.prototype, {
      '$patchFields': function(fields, success, error) {
         var self = this;
         patch = self.generatePatch(fields);
         var result =
            { id: },
         return result.$promise || result;
      generatePatch: function(fields) {
         var self = this;
         if(!angular.isArray(fields)) {
            fields = [ fields ];

         return fields.reduce(function(result, fields) {
            setFieldByPath(result, field, self.getFieldByPath(field));
            return result;
         }, {});
      getFieldByPath: function(path) {
         var ret = angular.toJson(json);
         var paths = path.split('.');
         for(var i = 0; i < paths.length; ++i) {
            if(angular.isUndefined(ret[paths[i]])) {
               return undefined;
            } else {
               ret = ret[paths[i]];
         return ret;

   function setFieldByPath(obj, path, value) {
      var paths = path.split('.');
      var setOn = obj;

      for(var i = 0; i < paths.length - 1; i++) {
         var path = paths[i];
         if(!angular.isUndefined(setOn[path])) {
               angular.isObject(setOn[path]) &&
            ) {
               setOn = setOn[path];
            } else {
               throw new Error(
                  'Path ' +
                  path +
                  ' has an item that is not an object'
         } else {
            setOn[path] = {};
            setOn = setOn[path];

      if(!angular.isFunction(setOn[paths[paths.length - 1]])) {
         setOn[paths[paths.length - 1]] = value;
      } else {
         throw new Error(
            'Cannot set value at ' +
            path +
            ' since it would overwrite a function'

   return Product;


Testing Angular $q Promises

This is a quick reference to testing Angular’s $q promises.

Values of promise.$$state.status
Value State
0 Outstanding
1 Resolved
2 Rejected