A little over five years have come and gone since I wrote Questions About Tooling, and as I was paging through my site  —  handling my quarterly SSL certificate reissuance, because it's easier spending five minutes re-running the same certbot command every three months than 20 minutes figuring out how to automate it  —  I found this post pretty interesting to look back on.

1. Yarn vs npm

I completely switched to Yarn shortly after writing the post, and I loved the experience. But I've recently come back to npm after the yarn 2.0 release and the sheer number of steps required to migrate legacy codebases  —  of which I maintain many  —  to their PnP architecture. I keep forgetting to add run to my commands, but other than that npm is just fine.

2. webpack vs browserify

Well, webpack definitely won this one. I still maintain a few repos with browserify-based scripting, but any new project will have webpack enabled from the start. The experience with webpack is a lot better now too, as this will typically be abstracted away from you via create-react-app, create-next-app, create-remix, or whatever scripts you're using to bootstrap your platform of choice. All of the advantages of webpack with none  —  well, less  —  of the googling to figure out what packages you need to install for your code to compile.

3. VS Code vs Atom

I'm still very happy with VS Code  —  to the point that we're now using it's core architecture to power the SuperHi Editor  —  though I no longer put as much effort into theming as I once did. I'll set the typeface to 16px IBMPlexMono-Regular, choose a nice theme with light and dark modes and leave everything else. Once I've added Prettier, GitLens and ESLint of course.

4. Typescript vs Flow

Another resounding win in the column, this time for Typescript. The tooling's gotten even better and the team are constantly adding new features and various improvements to the way that it works. At this point I'm uncomfortable whenever I have to write vanilla JavaScript, as I need all of the various safeties and niceties of Typescript to protect me from myself.

We'll only be writing a small application that toggles a single property in the store, but everything here gives you the toolset you need to expand upon and build a real-world application with. In fact, most of the code is taken directly from the new SuperHi Editor that I've been working on.

We'll dive in:

import { Record } from 'immutable'

export type AuthStatus = 'LOGGED_IN' | 'LOGGED_OUT'

export interface AuthStateProps {
  status: AuthStatus
}

const initialAuthState = Record<AuthStateProps>({
  status: 'LOGGED_OUT'
})

export class AuthState extends initialAuthState implements AuthStateProps {}

export const LOGGED_IN = 'LOGGED_IN'
export const LOGGED_OUT = 'LOGGED_OUT'

interface LoggedInAction {
  type: typeof LOGGED_IN
}

interface LoggedOutAction {
  type: typeof LOGGED_OUT
}

export type AuthActionTypes = LoggedInAction | LoggedOutAction

The first file is auth/types.ts, it's very similar to the Redux docs on Type Checking Actions & Action Creators but deviates with the use of immutable.js, we're using Immutable in the SuperHi Editor because it's noticeably faster than working with large JavaScript objects. The Redux docs on Using Immutable.JS with Redux are again a good starting point but we're ignoring their best practices  —  such as using a Higher order Component to convert Immutable objects to JS objects  —  in the name of speed, everything is an Immutable object  —  typically Records  —  and we'll work with them directly.

Why Immutable Records? Because they're much "safer" than Immutable Maps. Basically, a record allows us to guarantee the keys, so when we type record.get('filenamr') instead of record.get('filename') TypeScript will tell us we've done something wrong.

First we set up the plain object  —  or "props"  —  for the state:

export interface AuthStateProps {
  status: AuthStatus
}

Then we create the initial state given the default props:

const initialAuthState = Record<AuthStateProps>({
  status: 'LOGGED_OUT'
})

And then we build a class that extends this Record, allowing us to build records with new AuthState(props):

export class AuthState extends initialAuthState implements AuthStateProps {}

Typically in this class we'd also want to add the props  —  in this case there's just one: public readonly status!: AuthStatus  —  but if you're using create-react-app  —  as we do in the SuperHi Editor  —  then you're compiling your TypeScript with Babel, not with TypeScript itself and this will lead to a runtime error of Cannot set on an immutable record.

Now for auth/actions.ts

import { LOGGED_IN, LOGGED_OUT, AuthActionTypes } from './types'

export const loggedIn = (): AuthActionTypes => ({
  type: LOGGED_IN
})

export const loggedOut = (): AuthActionTypes => ({
  type: LOGGED_OUT
})

This is really simple and  —  for now!  —  exactly the same as the Redux docs. We'll come back to actions when we implement thunk actions later.

For auth/reducers.ts

import { AuthState, AuthActionTypes, LOGGED_IN, LOGGED_OUT } from './types'

export const initialState = new AuthState()

export default (state = initialState, action: AuthActionTypes) => {
  switch (action.type) {
    case LOGGED_IN:
      return state.set('status', 'LOGGED_IN')
    case LOGGED_OUT:
      return state.set('status', 'LOGGED_OUT')
    default:
      return state
  }
}

Pretty similar here too, though we build an initialState with our AuthState Record and export it for later. Because we're using AuthActionTypes here TypeScript will know exactly what payload to expect with each case, although in our demo it'll always be empty.

We'll also add auth/selectors.ts so we can grab that status as needed:

import { createSelector } from 'reselect'
import { AuthStatus } from './types'
import { AppState } from '../reducers'

export const selectAuthStatus = createSelector(
  (state: AppState) => state.getIn(['auth', 'status']),
  (authStatus: AuthStatus) => authStatus
)

We can use createSelector here as authStatus is going to be a standard JS string but if we instead wanted to select the entire auth slice of the store we'd need to do:

import { createSelectorCreator, defaultMemoize } from 'reselect'
import { is } from 'immutable'
import { AuthState } from './types'
import { AppState } from '../reducers'

const createImmutableSelector = createSelectorCreator(defaultMemoize, is)

export const selectAuth = createImmutableSelector(
  (state: AppState) => state.get('auth'),
  (auth: AuthState) => auth
)

This allows us to use Immutable's is function to compare two immutable objects  —  in this case the previous auth state and the new auth state  —  guaranteeing whether or not they're the same and making sure we don't re-render the React app accidentally.

Our root reducers.ts file is where things get a little messy:

import { Record } from 'immutable'
import { Reducer } from 'redux'
import { ThunkDispatch as TDispatch, ThunkAction as TAction } from 'redux-thunk'
import { combineReducers } from 'redux-immutable'
import { BatchAction } from 'redux-batched-actions'
import authReducer, { initialState as initialAuthState } from './auth/reducers'
import { AuthState, AuthActionTypes } from './auth/types'

interface AppStateProps {
  auth: AuthState
}

const initialAppState = Record<AppStateProps>({
  auth: initialAuthState
})

export class AppState extends initialAppState implements AppStateProps {}

export type AllActionTypes = AuthActionTypes | BatchAction

export type ThunkDispatch = TDispatch<AppState, null, AllActionTypes>

export type ThunkAction = TAction<void, AppState, null, AllActionTypes>

const rootReducer = combineReducers({ auth: authReducer })

export default (rootReducer as unknown) as Reducer<AppState, AllActionTypes>

The first thing we do is set up AppState just as we did with AuthState, using the initialAuthState we defined in auth/reducers.ts

We export a union of all action types, this includes the AuthActionTypes  —  and as any other action types you might've used  —  as well as the special BatchAction type, allowing us to safely dispatch batchActions too. We also export a ThunkDispatch built up of the AppState, void  —  because we're not using any extra arguments with redux-thunk  —  and AllActionTypes as well as ThunkAction built the same way, with void as the first type argument as we won't be returning anything from our thunk actions.

Finally we build the rootReducer and then, because redux-immutable only allows us to use a Map here, we lie to TypeScript  —  and ourselves  —  and say this is actually a Reducer of AppState and AllActionTypes. This will give us type-checking for keys in the store even though in practice we don't have these guarantees, the API for Immutable Maps and Records are very similar so we can get away with it here.

And then in store.ts:

import { createStore, applyMiddleware } from 'redux'
import { enableBatching } from 'redux-batched-actions'
import thunk from 'redux-thunk'
import reducers from './reducers'
import middlewareReducer from './middleware/reducers'

const middleware = applyMiddleware(thunk, middlewareReducer)

const store = createStore(enableBatching(reducers), middleware)

export default store

This looks pretty much as you'd expect, but wait, where did that middleware come from? We'll have a look at how to use middleware with all of this:

import { MiddlewareAPI } from 'redux'
import { AllActionTypes, ThunkDispatch } from '../reducers'

export default ({ dispatch }: MiddlewareAPI<ThunkDispatch>) => (next: ThunkDispatch) => async (
  action: AllActionTypes
) => {
  next(action)
}

It's actually not too bad! We're using our previously declared ThunkDispatch here to make sure we can dispatch on both normal actions and thunk actions. We could even set up a switch statement here and get the same level of safety we did in auth/reducers.ts. Note if you add your own special middleware-only actions you're also going to want to set up middleware/types.ts and middleware/actions.ts files for these and add the actions to AllActionTypes in reducers.ts.

Speaking of auth, we'll add a thunk action to auth/actions.ts:

export const handleLogIn = (payload: {
  username: string
  password: string
}): ThunkAction => dispatch => {
  // do stuff with the username and password, typically done in middleware for
  // side-effecting stuff like this.
  dispatch(loggedIn())
}

We'll use our ThunkAction here from reducers.ts to keep TypeScript happy, and if we use this in a component:

import React from 'react'
import { bindActionCreators } from 'redux'
import { connect } from 'react-redux'
import { AppState, ThunkDispatch } from '../store/reducers'
import { selectAuthStatus } from '../store/auth/selectors'
import { handleLogIn } from '../store/auth/actions'

const mapStateToProps = (state: AppState) => ({
  authStatus: selectAuthStatus(state)
})

const mapDispatchToProps = (dispatch: ThunkDispatch) =>
  bindActionCreators({ handleLogIn }, dispatch)

type Props = ReturnType<typeof mapStateToProps> & ReturnType<typeof mapDispatchToProps>

const Index = ({ authStatus, handleLogIn }: Props) => {
  return authStatus === 'LOGGED_IN' ? (
    <div>Logged In</div>
  ) : (
    <div
      onClick={() =>
        handleLogIn({
          username: 'artcommacode',
          password: 'this-is-not-my-password'
        })
      }
    >
      Log In
    </div>
  )
}

export default connect(mapStateToProps, mapDispatchToProps)(Index)

Here it's again mostly the same as you'll see in docs, the trick is using AppState and ThunkDispatch from our root reducer file in mapStateToProps and mapDispatchToProps, and ReturnType<typeof mapStateToProps> & ReturnType<typeof mapDispatchToProps> to get the actual shape of our props for the component. You can also extend Props if there's any props being passed in from above. We're also using our selector from auth/selectors.ts to make sure our data is memoised.

One final gotcha is if you're importing your store in a file and using that directly  —  say you're trying to do something right up top before you add your <Provider> wrapper  —  and dispatching a thunk action. Instead of store.dispatch(action()) you'll want to do (store.dispatch as ThunkDispatch)(action()) so TypeScript knows what you're trying to achieve here.

We're also using Apollo in the SuperHi Editor and I considered adding detailed instructions on getting that set up as well, but in practice it depends on what you need. The easiest method is to throw out Redux and just use Apollo, but although Apollo gives you an internal cache it doesn't have the tools  —  see above for the sheer number of them we're using here!  —  for working on state like you would in Redux.

Instead, the solution we came up with was to implement Apollo as a Redux middleware which allows us to use Apollo's client.watchQuery, client.mutate and client.queryManager.startGraphQLSubscription methods directly to talk to our API. It looks something like this:

switch (action.type) {
  case APOLLO_WATCH_QUERY: {
    const { name, query, variables, onResult, onError } = action.payload
    try {
      const observable = client.watchQuery({ query, variables })
      const { data } = await observable.result()
      if (data && data[name]) {
        dispatch(onResult({ [name]: data[name] }))
      } else {
        throw new Error(`${name} wasn't found in the response`)
      }
    } catch (error) {
      dispatch(onError({ error: error.message }))
    }
    break
  }
}

1. Should I use Yarn over npm?

Not just yet. Heroku  —  my deployment method of choice  —  say they support Yarn but my builds often fail. Added to this: Yarn can't be installed the recommended way when using nvm, global installs (unfortunately sometimes necessary) don't work as expected and the promised speed increases over npm aren't noticeable when my 80KB/s internet connection is the bottleneck. As such I'm putting Yarn to the side for a little longer.

2. Should I use webpack over browserify?

Probably not. From spending time in Slack and on Twitter you'd expect the gains to be massive but although my browserify scripts occasionally end up looking like the below there's many methods to clean that up. Going back to hundreds of lines of JavaScript to control my builds feels too much like a return to gulp.

"NODE_ENV=production browserify ./src/App.js -t [ babelify --presets [ es2015 stage-2 react ] --plugins [ transform-flow-strip-types transform-class-properties ] ] | uglifyjs > ./public/js/bundle.js"

If you use browserify and feel like you're missing out or if you use webpack and want to know how to do bundle splitting, loaders, source maps or more then I recommend checking out substack's great post "browserify for webpack users".

3. Should I use VS Code over Atom?

Absolutely! It's less resource intensive, much faster and has a great set of defaults including: tooltips, css completion that actually works, debugging, great git integration and many neat tricks. Plus if you don't mind hacking it up a little you can have a beautiful editor as well.

4. Should I use TypeScript over Flow?

Maybe? Although TypeScript's known to be unsound they make a good case for that and in my experience Flow has many issues with soundness too. VS Code's TypeScript tooling makes it an obvious winner when using the editor but TypeScript also beats out Flow in the shear number of typings available for external libraries and hasn't once yet told me that an EventEmitter isn't an EventEmitter.

Either way they're both a good midway point between untyped, vanilla JavaScript and something strongly typed like Purescript. I'm currently choosing between them on a project-by-project basis and usually use Flow for React and the frontend and TypeScript on the server.

Edit 03/2022: I have some more recent thoughts around JavaScript tooling here

"a monad is just a monoid in the category of endofunctors"
"monad is just a monoid in the category of endofunctors"
"is just a monoid in the category of endofunctors"
"just a monoid in the category of endofunctors"
"a monoid in the category of endofunctors"
"monoid in the category of endofunctors"
"in the category of endofunctors"
"the category of endofunctors"
"category of endofunctors"
"of endofunctors"
"endofunctors"

How would you handle the transformation? My first idea was to use two folds (or reduce in JavaScript speak):

const permutations = (str) => {
  const words = str.split(' ')
  return words.reduce((p, _, i) => {
    return p.concat([words.reduce((s, word, j) => {
      return j >= i ? s + ` ${word}` : s
    }, '').trim()])
  }, [])
}

Here I'm splitting the string into an array of words and folding over it twice to build an array of strings of words. However the first fold is basically xs.reduce((ys, x) => ys.concat([fn(x)]), []) and is equivalent to xs.map(fn), meaning the above can be rewritten as:

const permutations = (str) => {
  const words = str.split(' ')
  return words.map((_, i) => (
    words.reduce((s, word, j) => (
      j >= i ? s + ` ${word}` : s
    ), '').trim()
  ))
}

Which is already a little easier to understand. But I don't need that second fold at all, as instead of taking an array of words, finding all words past a certain index and concatenating them into a string it's much neater to simply slice the array at that index and join it back into a string. If I re-rewrite the function I get:

const permutations = (str) => {
  const words = str.split(' ')
  return words.map((_, i) => words.slice(i).join(' '))
}

Much better! And seeing as JavaScript gives us the original array as the third argument to map I can take the whole thing down to a tweet-sized chunk:

const permutations = (str) => (
  str.split(' ').map((_, i, words) => words.slice(i).join(' '))
)

I have a habit of jumping straight to folds as the solution to list problems. Seeing as they're the root of all iteration methods (it can be seen above how I accidentally implemented map and the same can be done for filter, some, find and all the others) the answer won't be wrong, but it will be overly complicated. I'm quite happy with how easy it is to read the solution when compared to my initial attempt.

Recently I had the pleasure of working with Hassan Rahim on Geordie Wood's new website.

It was fantastic. Working with Hassan's clean, intelligent designs for a man who's photographed the greats  —  from Obama to Gucci Mane  —  was an inspiration. Hassan's attention to detail is immense and after his carefully labelled Dropbox folders and intensely annotated Invision boards I may never be happy to go back to the ubiquitous, industry standard PDF…

There were three main requirements for the project; a near instantaneous front end with the highest possible visual fidelity and an easy to use backend.

Now, these may seem fairly obvious requests for a photographer's portfolio site but they're not as common as you'd expect so here's a little about how I went about it.

Starting with the frontend, working with large images meant I had to be sure I was sending correctly sized files to each and every client. Previously I'd resize the images on upload to the CMS, but this slows down the editing process and leaves me with just a few sizes to choose from.

So this time I turned to Imgix, and all I had to do was point it to an Amazon S3 bucket and make a request with the filename and dimensions of the image (calculated based on the size the image is to be shown at, the screen size and window.devicePixelRatio). I rounded all sizes to the nearest 50px to make sure I'd hit Imgix's cache as often as possible, as a cache hit takes only a few milliseconds but with a miss it can be over a second as we wait for Imgix to resize the image before sending it back.

As an aside, I'm only using a few libraries on the frontend  —  React and React Router are the two big ones  —  and all my code's written in what I've taken to calling ES6+ (ES6 with a few neat ES7 extras such as async and await) and compiled with Babel.

With the image sizes sorted I had to make sure they were loaded as quickly as possible. For the desktop I went with a very aggressive caching strategy that loads all of the slides in the background one-by-one. Though I made sure to take the first slide out of each project and loaded those in immediately so they were ready when the user interacted with the homepage.

For mobile it's a little different as I couldn't take the desktop strategy because at best it noticeably slowed things down and at worst it crashed the tab entirely (something that happened a lot on earlier iPads as low internal memory and large images aren't a good mix). So instead the site waits until the user hits a slide and simply loads in that slide and the one immediately after it. It's not a perfect solution but it still feels rapid and doesn't cause any slow-downs.

The backend is very different, while the frontend is rendered almost entirely in the browser the backend is a more typical website. I use Express (I am a member of the organisation and an operator in the IRC channel #express after all), Postgres and a relative newcomer to the Node.js ORM scene: Objection.js. Prior to this I'd been using Bookshelf in all my projects but was increasingly dissatisfied with the way it forces a Backbone-like structure on you and felt that it made too many things (such as validation and nested relations) harder to implement than they should've been.

The Objection documentation is also a lot more thorough than Bookshelf's and an example repo showing you how to write a basic site in ES5, ES6 and ES7 is an added bonus. Seeing as I was compiling everything anyway I took the ES7 route, allowing me to write code like:

router.get('/', async (req, res, next) => {
  const projects = await Project.query().orderBy('position')
  res.render('projects/index', {projects})
})

and:

const project = await Project.query()
  .where('id', +req.params.id)
  .eager(
    '[slides(orderByPosition), slides.images(orderByPosition)]',
    {orderByPosition}
  )
  .first()

(Objection's eager queries make nested relations absolutely painless)

The main part of the backend is the drag-and-drop slide editor:

With this Geordie can simply upload images and drag and drop them into the layout grid. They fall where they're dropped  —  snapped to the nearest column  —  and a click can set their width. I used the standard HTML5 drag-and-drop API for this:

const onDragStart = e => {
  const {offsetX, offsetY} = e
  e.dataTransfer.effectAllowed = 'move'
  e.dataTransfer.setData('text/plain', JSON.stringify({offsetX, offsetY}))
}

const onDrop = e => {
  e.preventDefault()
  const {offsetX, offsetY} = JSON.parse(e.dataTransfer.getData('text'))
  // ...
}

The rest is just some maths to figure out which column we're on and then sending this data to the server. There's only two fields needed in the database for this; columnsLeft (the column number the image starts at) and columnsWide (the width of the image in columns). Everything else is extrapolated from this and our 16 column grid.

And that's the majority of it!

Thanks to Hassan and Geordie for being such a delight to work with, thanks ​again to Eric Hu for setting me up with them and thanks to the snek team on Slack for helping me brainstorm the best way to lay out the thumbnails.

If you have any questions about this project, or any other projects, get in touch with me via twitter or at ryan@artcommaco.de.

Update 06/16: paintingid.com has been turned off, but you can visit an archive of the site at paintingid.artcommaco.de.

I recently had the pleasure to work on a website for the NYC artist Brendan Smith with designers Harry Gassel and Seth Hoekstra and 3D Illustrator James Orlando.

One of those short-notice jobs that always has the potential to become a nightmare it nontheless turned out to be a great time thanks to the professionalism of my main-point-of-contacts Harry and Seth. However, three.js was another matter entirely. A mess of horrible documentation, awkward anti-patterns and hundreds of mutable variables it took me a week before I could even display a painting on the screen and when I did it looked something like this:

After that it was a matter of figuring out rotations, positioning, texturing, hooking up controls and how to swap out colours and paintings on the fly. I then discovered how tricky it is to light a scene to show anything from white to bright purple to black. On the advice of a friend I ended up going with two lights, a very dark blue-black ambient light and a very bright yellow-white directional light.

Despite the pains of three.js it was great to have the opportunity to learn some new skills.

Overall I'm moving towards simpler, more focused tools that I can combine to produce smaller, more easier to understand packages.

I recently replaced my 100 line gulp file with 10 lines of npm directives which have already proven to be simpler and more reliable. 

Not happy with the hacks that were piling up in my gulpfile.js I started looking at the alternatives. Makefiles are too magical for me but I haven't ruled them out completely. For anybody interested in using them I highly recommend James Coglan's post "Building JavaScript projects with Make".

Instead I found substack's post "task automation with npm run" and have liberally copied from it. Because this current project is a static site and not one of my usual Express applications I've also added Jade to convert my .jade files to HTML and http-server to serve the site. 

So I'm going to show you what I do instead. Lets get rid of that app.use(require('express-messages')()) and replace it with:

app.use(function (req, res, next) {
  var flash = req.flash()
  res.locals.messages = Object.keys(flash).reduce(function (messages, type) {
    flash[type].forEach(function (message) {
      messages.push({type: type, message: message})
    })
    return messages
  }, [])
  next()
})

What we're doing here is taking the object of arrays generated by connect-flash and turning it into an array of objects so I don't have to do any fiddly logic to include it in my templates. If I'm using Jade (my personal favourite) I can do this: 

ul.messages
  each message in messages
    li(class="#{message.type}")= message.message

and if I'm using EJS I do this:

<ul class="messages">
  <% messages.forEach(function (message) { %>
    <li class="<%= message.type %>"><%= message.message %></li>
  <% }) %>
</ul>

Now instead of asking express-messages to concatanate some strings into HTML for us to include as a function in our template we get a nice messages array on the locals object that we can include wherever and however we want.

It also serves as a gentle introduction to Haskell's Type Annotation which I personally find useful even when not working in Haskell, or in languages without proper typing at all! (JavaScript I'm looking at you)

Collection.findAndPopulate({
  title: this.controller
}, function (error, collection) {
  if (error) return callback(error);
  collection.pages = _.filter(collection.pages, function (page) {
    return _.first(page.title.split(' ')) === req.params.year;
  });
  callback(null, collection.pages);
});

Pretty cool right? Underscore is great when working with lists. However, a few hours after pushing the new code to production I realised (while in the shower) I could've skipped the Underscore and just done:

Page.findAndPopulate({
  title: new RegExp('^' + req.params.year),
  collection: this.controller
}, callback);

• A History of Haskell - Being Lazy With Class
• Design of LISP-Based Processors (AIM-514)
• Design Principles Behind Smalltalk
• Kqueue - A generic and scalable event notification facility
• Making reliable distributed systems in the presence of software errors (Erlang, Armstrong)
• Organizing Programs Without Classes
• PC Assembly Language (Paul A. Carter, 2001-2004)
• Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part 1 (John McCarthy, lisp, 1960)
• RRB-Trees - Efficient Immutable Vectors
• Synthesis - An Efficient Implementation of Fundamental Operating System Services
• The GrAIL language and operations
• The Implementation of Lua 5.0
• x86-64 Machine-Level Programming (CMU)

Of these already great papers, Joe Armstrong's thesis “Making reliable distributed systems in the presence of software errors” is my stand-out favourite. Written in a clear, concise manner it defines the problems he and his team were facing and goes on  —  through it's nearly 300 pages  —  to solve each of them. I could (and probably should) dedicate an entire post to this paper alone.