Vinicius Gomes

Event Driven Architectures & Serverless

2018-07-23T01:36:00+00:00

After a 2.5 years journey building an event-driven services platform for a large financial organization, I’ve recently joined a new team in the middle of their on-going migration to the public cloud. One of the things we’ve been exploring is the Serverless model in order to release software with close to none operations effort.

Turns out there are significant commonalities between EDAs (Event-Driven Architectures) and Serverless which can lead to interesting advantages from the architectural point of view when applied to the appropriate business context. That was the topic of my recent presentation at the North America XConf.

Event Driven Architectures

An EDA is a distributed system where the communication between the functional components is asynchronous and implicit - in opposition to the more traditional service-oriented architectures where the communication is often times synchronous and explicit. EDAs rely on a message broker (AKA Event Bus) responsible for integrating producers and consumers.

Elements of an EDA

There are different flavors of Event Driven Architectures, but the one I’d like to focus on is based on Event Sourcing (ES) and Command Query Responsibility Segregation (CQRS).

Essentially, a CQRS application has two separate data models: Command (writes) and Query (reads). Although both models are part of the same bounded-context they are independent and communicate implicitly through the Event Bus. The Command side holds the event-sourced state of the application while the Query side maintains an aggregated projection of that state.

Internal structure of a ES/CQRS application

Overall, the benefits you’re likely to get when adopting an ES/CQRS EDA are:

Low coupling. Services don’t communicate explicitly so that there is no direct dependency between them. It doesn’t really matter which service publishes an event as long as the consumer is able to deserialize the message from the Event Bus.
Resilience. Services proactivelly consume events of their interest. As a consequence, they’re always aware of latest valid global state. That helps to avoid cascading failures when a given service goes down.
Performance. All data is aggregated asynchronously so that there is no internal ad-hoc communication when a client request comes in.
Immutability. As far as data goes, the source-of-truth is the Event Store - an immutable, append-only database. There is no current mutable state.
User intent. Modeling changes as typed Commands often gives us more expressive power than using HTTP methods against resources directly.
Chronology. With the Event Store we record every intermediate state of a user journey, not only the final state. That’s a powerful tool for UX analysis.
Audit Log by design. Keeping an audit log is a quite strong requirement for certain domains. That comes for free when you have an Event Store.
Debugging. Having access to the event history allows developers to better understand and debug the system behavior, specially when it comes to distributed transactions.

Check out my QCon talk from last year to get the details on this approach.

Serverless

According to Mike Roberts’ definition, Serverless can be described as an approach to build and deploy applications with low operations cost through the combination of BaaS and FaaS components.

BaaS or Backend-as-a-Service refers to the infrastructure components managed by a cloud provider. Examples are AWS S3, AWS DynamoDB, Google BigQuery.
FaaS or Function-as-a-Service refers to execution environments pre-defined and managed by a cloud provider. Examples are AWS Lambda, Google Cloud Functions, Azure Functions.

The Serverless model aims to minimize configuration management and server provisioning up to the point where your cost of operations is close to zero. In order to accomplish that, you rely heavily on the offerings of the cloud provider of choice. The main benefits of this approach are:

Abstraction. When you work with VMs you abstract the hardware; when you work with containers you abstract the kernel; when you work with cloud functions you abstract the execution environment, so that you can focus 100% on the application logic.
Deployment. Your cloud provider fully manages deployments so you don’t need to worry about downtimes or strategies like Blue/Green or rollbacks.
Scalability. Functions would dynamically scale by default without any custom configurations or infrastructure setup.
Monitoring. Your cloud provider gives you access to logs, metrics, and alerts around your functions without the need of setting up monitoring tools yourself.
Cost. The FaaS billing model is optimized for usage, so that you don’t get billed for idle time.

There is another important aspect which can not exactly be considered a benefit or a disadvantage: the granularity of a function. When it comes to Microservices we tend to think of a bounded context with correlated functionality running on its own process. A function, on the other hand, tends to be more granular. It often represents a single self-contained functionality running on a independent process. We can think of a service in Serverless as a combination of multiple functions and infra components. Although this granularity might enable more flexibility on deployments - as you can update parts of a service independently - it also adds more complexity when it comes to integration/contract testing. I plan to discuss that further in a future post.

Event-Driven Serverless

Turns out a Serverless application is fundamentally an Event-Driven application. Functions are triggered by events - either from BaaS components or from other functions through a message broker. Examples of triggers on AWS:

A file uploaded to an S3 bucket;
Inserts on a DynamoDB table;
A message published to an SNS topic;
A CloudWatch alert.

Moreover, there is no direct communication between functions which makes them loosely coupled and more resilient - the same way as it happens in an EDA. (Even when it comes to HTTP, the communication actually happens through events produced by a gateway of some sort, like the AWS API Gateway.)

Moving further, we can apply the CQRS/ES pattern when modeling services as functions. Here’s an example using AWS abstractions:

Internal structure of a Serverless ES/CQRS application

The red lines on the diagram represent new data being written; the green lines represent existing data being read. Similarly to the non-serverless version, we also have the Command side on the top (Commands, Event Store, Event Publisher) and the Query side on the bottom (Event Listener 1/n, Query DB, Queries) as well as an Event Bus.

As you may have noticed, our service is composed by 5 functions and 4 BaaS components described below.

The API Gateway represents the entry point for both command and query requests.
Both the Event Store and the Query DB are implemented with DynamoDB.
The Event Bus is implemented as a set of SNS topics.

As far as the data flow goes, we basically have a Commands function handling incoming command requests and inserting new domain events to the store. Those new events would be broadcasted by the Event Publisher function to the bus. All consumers are notified and - asynchronously - would update the query projections on the Query DB. The last Queries function simply responds to incoming queries from the API Gateway.

Check out a sample implementation of this idea on my Github: https://github.com/vvgomes/serverless-restaurant

Concluding Thoughts

Here are a few takeaways I believe are important to keep in mind in case you are considering to adopt a Serverless EDA approach.

Cost of operations - If operations is a significant cost to your organization, moving to the Serverless model can be an interesting financial option, as you’d be able to transfer almost all the server provisioning and configuration management burden to the cloud provider.

Vendor Lock-in - On the other hand, once you adopt a particular cloud provider, you are pretty much restricted to the offerings of that provider. For instance: there is no BigQuery in AWS, there is no RedShift in GCP, etc.

Customization - Another point to consider is the level of customization you’ll need to introduce on infrastructure components in order to support the peculiarities of your problem domain. BaaS solutions are usually not very customization-friendly.

Predictability - Although the FaaS billing model is pretty interesting for situations where you don’t really know what kind of traffic to expect, for scenarios where the load is reasonably high and constant it is probably cheaper to just keep a few robust IaaS compute engines running permanently instead.

Nature of the problem - Overall, going Serverless - and even more - going Event-Driven is something that might not be applicable to certain domains. There are situations where using those patterns just doesn’t make sense. After all, there is no silver bullet.

When Microservices Meet Event Sourcing

2017-05-29T03:55:00+00:00

For the last two years my team has been working in the development of a distributed services platform for a major US-based bank. This platform is intended to onboard a few products in the beginning but gradually grow as legacy products are also migrated to it. There are several challenges and cross-functional requirement involved like extensibility, audibility, analytics, etc, which have naturally driven us to move towards an Event Sourcing and CQRS design.

After the successful release of the first few products in production running on this architectural style, we felt it was a good time to share our learnings with the world. Recently I had the honor to present our journey twice: on April at the O’Reilly Software Architecture Conference and on May at the ThoughtWorks Tech Talks NYC. The feedback from the audience was above expected, both with skepticism and enthusiasm. Checkout the presentation video:

The slides are published here: speakerdeck.com/vvgomes/when-microservices-meet-event-sourcing
The demo source code is here: github.com/vvgomes/restaurant-oreillysacon

Frequently Asked Questions

Many people came to me and to people from my team with questions about this architectural style. Here are the most popular ones:

How should clients deal with Eventual Consistency?

Eventual Consistency is one of the accidental complexities of event-driven CQRS applications. The problem basically consists of the fact command execution is usually implemented in the fire-and-forget style which means there are no guaranties about when the result is available for querying.

A possible answer for that question can be the separation between the command execution result and the data result. Right after the client sends a command to be processed, the CQRS application must - at least - synchronously respond with a result. This is either success (2XX status code) or a failure (> 4XX status code). Besides that, it is a good practice to return an URI (in the Location HTTP header, for instance) that guides the client where to find the data result. As far as querying the data itself, there are a few known options:

Polling - the client code keeps polling the URI returned by the command execution until the data is ready.
Websockets - the client passivelly waits for the server to send the data back through a websocket connection.
Optimistic Rendering - based on a successful command execution, the client can optimistically render the results without actually querying the backend.

In fact, our experience shows that in the vast majority of the cases the data will be available for querying pretty much at the same time as the command request is done, which means that the client will mostly be able to just query the data right after it is done sending the command. Anyway, in case you know about alternative options to deal with Eventual Consistency from the client side please let me know.

How do you deal with changes in the event payload structure?

Since events are immutable and the event-store append only, there is no room for database migrations in Event Sourcing. Therefore, when it comes to schema changes we need to define mapping functions which will translate older to newer version of events. This technique is known as Event Upcasting.

On the other hand, we don’t want to break consumers after changing the payload structure of an event. That can be accomplished by having Contract Tests running in your CI pipeline. In this context, Contract Tests will pretty much verify whether consumers are still able to successfully deserialize the most recent version of a given event.

What about performance?

Over time, the amount of events in the event-store can grow to a point in which rebuilding the current state of the application becomes expensive. To minimize this problem we can persist intermediate states as Snapshots and then only replay the most recent events on top of it. Archiving very old events in a secondary storage and using snapshots permanently is also an option in extreme cases. Overall, managing an in-memory cache of the current state in a running application is a reasonable solution for most cases.

How can we integrate with non-event-driven systems?

When it comes to integrating event-driven applications with non-event-driven (internal or external) systems, there are a lot of different aspects to be discussed like real-time vs batch processing, communication protocols, resilience, SLAs, etc. Although that goes beyond my intended scope, you can still find most of the answers on that regard by watching this excellent presentation by my teammate @javatarz:

Next Presentations

I’ll be talking about this topic again soon in two more conferences:

6/21 Developer Week NYC
6/26 QCon New York

Dependency Injection in Javascript

2017-03-17T03:55:00+00:00

If you like to develop your Javascript applications in a functional style, often times you have to deal with Dependency Injection when it comes to unit-test functions that depend on other functions. Fortunately, with ES6 default parameters DI becomes quite simple. Let’s see an example.

Example: An API Client

Most client applications need to communicate with a remote server, So let’s build a simple API client module with a few functions. The end goal looks like this:

// an usual TODO app...

import * as apiClient from "./lib/api.client";

apiClient
  .addTodo({ text: "wash dishes" })
  .then(todo => console.log("New todo added:", todo))
  .catch(errors => console.log("Something went wrong:", errors));

//=> New todo added: { id: "1", text: "wash dishes", done: false }

// ...

apiClient
  .fetchTodos()
  .then(todos => console.log(todos))
  .catch(errors => console.log("Something went wrong:", errors));

//=> [ { id: "1", text: "wash dishes", done: false } ]

Our client is just a namespace of functions, each one representing a different HTTP request to a remote server. To send the actual request we intend to use fetch which returns a Promise wrapping the response. So the fetch function will be our dependency.

Starting With Tests

Let’s write a few unit test cases to drive the development of our functions. In order to test them in isolation we need to inject a fake fetch function as we don’t want to exercise the real HTTP requests.

// test/api.client.test.js
import * as client from "../lib/api.client";
import { expect } from "chai";
import { stub } from "sinon";

describe("api client", () => {
  let fakeFetch;

  beforeEach(() => {
    fakeFetch = stub();
  });

  describe("fetchTodos()", () => {
    it("retrieves all todos", () => {
      const fakeResponse = Promise.resolve({
        status: 200,
        json: () => Promise.resolve([{
          id: "1",
          text: "walk the dog",
          done: false
        }])
      });

      fakeFetch.withArgs("/api/todos").returns(fakeResponse); 

      return client.fetchTodos(fakeFetch).then(todos => {
        expect(todos).deep.eq([{
          id: "1",
          text: "walk the dog",
          done: false
        }]);
      });
    });
  });
});

We stubbed fetch to return an arbitrary response and passed as an argument to fetchTodos. That can be implemented as a default parameter.

// lib/api.client.js
import defaultFetch from "isomorphic-fetch";

export const fetchTodos = (fetch = defaultFetch) =>
  fetch("/api/todos").then(res => res.json());

We’re importing the real fetch implementation as the default argument value. This way, we keep the proposed interface unchanged. Let’s add a test for addTodo as well.

// test/api.client.test.js
import * as client from "../lib/api.client";
import { expect } from "chai";
import { stub, match } from "sinon";

describe("api client", () => {
  let fakeFetch;

  beforeEach(() => {
    fakeFetch = stub();
  });

  // ...

  describe("addTodo()", () => {
    it("creates a new todo successfully", () => {
      const requestData = match
        .has("method", "POST")
        .and(match.has("headers"))
        .and(match.has("body", JSON.stringify({ text: "wash dishes" })));

      const fakeResponse = Promise.resolve({
        status: 202,
        json: () => Promise.resolve({
          id: "2",
          text: "wash dishes",
          done: false
        })
      });

      fakeFetch.withArgs("/api/todos", requestData).returns(fakeResponse); 

      return client.addTodo({ text: "wash dishes" }, fakeFetch).then(todo => {
        expect(todo).deep.eq({
          id: "2",
          text: "wash dishes",
          done: false
        });
      });
    });
});

Here again, we pass the fakeFetch as an argument to addTodo. The implementation will be similar as the one for fetchTodos.

// lib/api.client.js
import defaultFetch from "isomorphic-fetch";

export const fetchTodos = (fetch = defaultFetch) =>
  fetch("/api/todos").then(res => res.json());

export const addTodo =  (todo, fetch = defaultFetch) =>
  fetch("/api/todos", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ text: todo.text })
  }).then(res => res.json());

At this point, we can remove the code duplication by moving the default parameter to the namespace level. For that, we’ll define a createApiClient function. Let’s change the tests accordingly.

// lib/api.client.test.js
import createApiClient from "../lib/api.client.";
import { expect } from "chai";
import { stub, match } from "sinon";

describe("api client", () => {
  let fakeFetch;
  let client;

  beforeEach(() => {
    fakeFetch = stub();
    client = createApiClient(fakeFetch)
  });

  describe("fetchTodos()", () => {
    it("retrieves all todos", () => {
      const fakeResponse = Promise.resolve({
        status: 200,
        json: () => Promise.resolve([{
          id: "1",
          text: "walk the dog",
          done: false
        }])
      });

      fakeFetch.withArgs("/api/todos").returns(fakeResponse); 

      return client.fetchTodos().then(todos => {
        expect(todos).deep.eq([{
          id: "1",
          text: "walk the dog",
          done: false
        }]);
      });
    });
  });

  describe("addTodo()", () => {
    it("creates a new todo successfully", () => {
      const requestData = match
        .has("method", "POST")
        .and(match.has("headers"))
        .and(match.has("body", JSON.stringify({ text: "wash dishes" })));

      const fakeResponse = Promise.resolve({
        status: 202,
        json: () => Promise.resolve({
          id: "2",
          text: "wash dishes",
          done: false
        })
      });

      fakeFetch.withArgs("/api/todos", requestData).returns(fakeResponse); 

      return client.addTodo({ text: "wash dishes" }).then(todo => {
        expect(todo).deep.eq({
          id: "2",
          text: "wash dishes",
          done: false
        });
      });
    });
  });
});

As you can see in the new version of the tests, we’re now injecting fakeFetch to createApiClient. That new function acts like a contructor: it builds the namespace and make the injected fetch available to the entire namespace scope. Let’s make that change to the application code.

// lib/api.client.js
import defaultFetch from "isomorphic-fetch";

const createApiClient = (fetch = defaultFetch) => ({
  fetchTodos: () =>
    fetch("/api/todos").then(res => res.json()),

  addTodo: todo =>
    fetch("/api/todos", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ text: todo.text })
    }).then(res => res.json()),
});

export default createApiClient;

With the refactoring above, our initial code snippet will have to change a little bit. Now we have to call first createApiClient instead of just importing all functions.

// an usual TODO app...

import createApiClient from "./lib/api.client";
const apiClient = createApiClient();

apiClient
  .addTodo({ text: "wash dishes" })
  .then(todo => console.log("New todo added:", todo))
  .catch(errors => console.log("Something went wrong:", errors));

//=> New todo added: { id: "1", text: "wash dishes", done: false }

// ...

apiClient
  .fetchTodos()
  .then(todos => console.log(todos))
  .catch(errors => console.log("Something went wrong:", errors));

//=> [ { id: "1", text: "wash dishes", done: false } ]

The code above represent a sample usage of our module. Notice that the calling code doesn’t have to pass fetch to createApiClient since we used a default parameter. Besides that, we are now able to add more functions to the module (like toggleTodo, for instance) and have fetch available to them.

Conclusion

ES6 default parameters enable functional DI in simple and elegant way. The approach demonstrated in this post suggests a new point of view on function arguments:

The regular arguments - the arguments which matter to the application code.
Dependencies - collaborator functions defaulting to the real implementation and stubbed in unit tests.

Check out the complete example code at github.com/vvgomes/es6-di-example.

We don’t need a Tech Lead

2016-11-28T09:15:00+00:00

When it comes to medium to large teams it is quite common the presence of a full-time tech lead responsible for important leadership activities such as:

Guiding the project technical vision;
Analyzing risks and cross-functional requirements;
Coaching less experienced people;
Bridging communication between stakeholders and the team.

In general, the person assigned as the tech lead is someone who has sound technical experience as well as strong communication skills. She will be accountable for the technical direction of the project as well as serve as the go-to person for cross-team interactions.

Do we need a Tech Lead?

Pat Kua wrote an article about two years ago in which he supports the idea that the presence of a tech lead is necessary for most software teams. In his words:

People argue against the role, claiming a team of well functioning developers can make decisions and prioritize what is important to work on. I completely agree with this position in an ideal world. Sadly the ideal world rarely exists.

Based on my observations and personal experience, I tend to disagree with that point of view in two ways:

Well functioning teams in which people share responsibilities are not rare.
When a team is not functioning well, assigning a tech lead can potentially make it worse.

Is the Tech Lead a workaround?

To say the tech lead is necessary because ideal conditions are rare is to say the tech lead is a workaround - not a root cause solution. That means the role is there just because circumstances are suboptimal. When a team is not functioning well there is probably a number of broken windows to be addressed. Tech leads could alleviate the consequences only.

In his article though, Pat Kua gives an example scenario in which a tech lead would be required:

Technical debates occur frequently in development teams. There is nothing worse than when the team reaches a frozen state of disagreement.

The Tech Lead has the responsibility to help the team move forwards… Facilitation and negotiation skills are invaluable assets to a Tech Lead.

Well, in a situation like that, what we probably need is a mediator. And that role could be played - on demand - by anyone with facilitation and negotiation skills. Pushing multiple skills expectations to a single person is not scalable; it overwhelms the person in charge and undermines the entire team.

Does the Tech Lead make it worse?

About four years ago, I wrote about the importance of people exercising different skill-sets and playing multiple roles in a team.

The most successful teams I’ve seen share the fact that people are multi-functional and self-managed, in a sense that every single member would be actively involved in a) helping the business to figure out and define requirements, b) directly work on the development of features, and c) directly work on testing related activities.

I strongly believe that can be extended to technical leadership activities as well. The main idea behind role rotation is risk mitigation. Regardless the tech lead level of experience, there is a variety of risks associated to the existence of the role. Those risks tend to increase in severity as the team becomes larger.

Risks to the Tech Lead

Overload of activities - too many expectations on a single individual.
Lack of focus on details due to the constant context switching.
Single point of failure - when the information is centralized on the tech lead.
Blame culture - the tech lead is usually the one accountable for failures.
Distance - when the tech lead is mostly away from the team.
Lack of the sense of peer - when the tech lead is not seen as a team member anymore.

Risks to the Team

Lack of collective ownership - when critical decisions are not collective.
Bottlenecks - when all the decisions must pass through the tech lead.
The Bus Factor - when the team cannot operate without the tech lead.
Impact on motivation - when team members feel unempowered and voiceless.

Besides that there is another psychological risk: in groups where two or more people have enough technical leadership skills, how to elect one of them as the tech lead without making the others feel depreciated?

In summary, those problems are a direct consequence of the full-time tech lead presence. And the eventual benefits the tech lead brings does not often pay back the burden of having to deal with the risks.

Leadership

Do we still need leadership in software teams? - We definitely do. But leadership as a skill-set rather than an individual role.

In a recent interview, Fred George suggests a seasonal and decentralized leadership model, in which leaders will naturally emerge based on the project needs at a given time.

When there are newcomers, we need a strong coach.
When there are architectural challenges, we need an experienced architect.
When there are internal conflicts, we need a mediator.
When there are external blockers or lack of resources, we need a concierge.
When we need to negotiate and integrate with other teams, we need an ambassador.

And those are just a few examples. It’s clearly not a good idea to setup expectations regarding all those capabilities on a single person (the tech lead). On the other hand, it seems more reasonable to build a team of people with complementary capabilities who can function effectively together.

Another perspective on decentralized leadership is the idea of Feature Leads (initially proposed by Pat Kua, and also discussed by my colleague Ryan Oglesby). In that approach, the technical leadership is broken down into slices of related functionality and/or ortogonal aspects of the system in development. It helps to minimize the risks by decentralizing leadership activities in multiple individuals.

Slicing the backlog and stakeholders by Feature Leads

Conclusion

So, do we really need a tech lead? - Maybe yes when the conditions are not ideal. But the tech lead almost certainly won’t be able to just turn the conditions into ideal. Teams operating in suboptimal circumstances in general have broken windows and blockers to be addressed. Many times you will find in XP Values the exact answers your team is looking for. Besides that, there is no silver bullet if you are not able to find the right people to work with you, so recruiting plays a big part here.

Overall, assigning a full-time tech lead in a team is risky and can potentially make things worse. Healthy and productive software teams most probably do not need a tech lead.

Elegant Validations in Javascript

2016-11-22T09:15:00+00:00

How do you model business rules validations? Although there seems to be a variety of approaches out there, some common abstractions are often present. Given an arbitrary data structure representing an entity in your domain, a validation algorithm will usually be composed by:

A group of predicates to check business rules against the entity.
A validation result which will either sign success or failure.
A collection of validation errors in case of failure.

It’s a quite simple protocol. But many people still keep on re-writting validation utilities using slightly variant approaches, specially in the Javascript world.

Throwing Runtime Errors

When it comes to validation errors, one of the most popular ways to handle them is by throwing runtime errors and/or exceptions. Particularly in Javascript, you’ll find people using native Error objects sometimes combined with .catch() calls in promises or just surrounded by try/catch blocks.

This approach however has two fundamental issues:

Validation errors are not necessarily exceptional. They are rather expected business scenarios. They are not in the same category as network/integration failures or invalid state caused by defects, for instance.
When you throw a runtime error, you skip the current thread execution, which makes it virtually impossible to accumulate all the validation errors together. See the example below.

  
const isPasswordLongEnough = password => {
  if (!password.length > 6)
    throw new Error("Password must have more than 6 characters");
}

const isPasswordStrongEnough = password => {
  if (!(/[\W]/.test(a)))
    throw new Error("Password must contain special characters");
}

const validate = password => {
  isPasswordLongEnough(password);
  isPasswordStrongEnough(password);
};

validate("foo");
// => Error: Password must have more than 6 characters.

validate("rosesarered");
// => Error: Password must contain special characters.

Notice that the first validate() call above should have resulted in two errors instead of just one.

Monadic Error Handling

Functional programmers familiar with Either will usually think about validations and error handling in terms of functions and data containers. In Javascript, one of the most elegant implementations can be found in the data.validation module from Folktale. (Here is an excellent article by @robotlolita on the topic.)

That module will give you a Validation container that is implemented by two subtypes: Success or Failure. They act like Right and Left in Either but customized to be explicit and handle error accumulation. See the example below (from the docs).

  
import Validation from "data.validation";
const Success = Validation.Success;
const Failure = Validation.Failure;

const isPasswordLongEnough = password =>
  password.length > 6 ?
    Success(password) :
    Failure(["Password must have more than 6 characters"]);

const isPasswordStrongEnough = password =>
  /[\W]/.test(password) ?
    Success(password) :
    Failure(["Password must contain special characters"]);

const validate = password =>
  Success(x => y => password)
    .ap(isPasswordLongEnough(password))
    .ap(isPasswordStrongEnough(password));

validate("foo");
// => Validation.Failure([
//      "Password must have more than 6 characters.",
//      "Password must contain special characters."
//    ])

validate("rosesarered");
// => Validation.Failure([
//      "Password must contain special characters."
//    ])

validate("rosesarered$andstuff");
// => Validation.Success("rosesarered$andstuff")

data.validation gives you (a) abstractions to represent success and failure, (b) the ability to accumulate errors, and (c) provides you with an interface to manipulate the results in a functional way.

The one inconvenience though is the fact that error accumulation requires a workaround. Take a look at the validate() function above: we need to call Success with two levels of function nesting because we are running two validation predicates. The explanation from the docs:

To aggregate the failures, we start with a Success case containing a curried function of arity N (where N is the number of validations), and we just use an .ap-ply chain to get either the value our Success function ultimately returns, or the aggregated failures.

That limits our ability to define a dynamic validation algorithm that would work with an arbitrary number of predicates. There is though a solution proposed in the article mentioned earlier. But I believe we can bridge that better by using one of my favorite JS libraries: Ramda.

Ramda For The Rescue

The first thing we would like to do now is to be able to dynamically define an auto-curried version of a N-ary function (N being the number of predicates) and then pass it to Success. We can accomplish that with Ramda’s curryN().

  
// ...

import { curryN } from "ramda";

const validate = password =>
  Success(curryN(2, (x, y) => password))
    .ap(isPasswordLongEnough(password))
    .ap(isPasswordStrongEnough(password));

validate("foo");
// => Validation.Failure([
//      "Password must have more than 6 characters.",
//      "Password must contain special characters."
//    ])

That’s the first step. It doesn’t give us much though. curryN() is defining an auto-curried version of the (x, y) => password function, so we don’t need nesting anymore. But we are still not dynamic, since the number of predicates is hard-coded. In order to fix that, we need an intermediate step: pass the validations being applied (in the.ap() calls) as an argument.

  
// ...

import { curryN, reduce } from "ramda";

const validate = (validations, password) =>
  reduce(
    (result, validation) => result.ap(validation(password)), // reducer fn
    Success(curryN(2, (x, y) => password), // initial value
    validations); // input

validate([isPasswordLongEnough, isPasswordStrongEnough], "foo");
// => Validation.Failure([
//      "Password must have more than 6 characters.",
//      "Password must contain special characters."
//    ])

Now, we are at least generic about the validations being applied to the initial Success state. For that we used Ramda’s reduce. The next step is to eliminate the hardcoded values in the curryN() call.

  
// ...

import { curryN, reduce, length, always } from "ramda";

const validate = (validations, target) =>
  reduce(
    (result, validation) => result.ap(validation(target)), // reducer fn
    Success(curryN(length(validations), always(target))), // initial value
    validations); // input

validate([isPasswordLongEnough, isPasswordStrongEnough], "foo");
// => Validation.Failure([
//      "Password must have more than 6 characters.",
//      "Password must contain special characters."
//    ])

The trick this time was to use the length of the validations array in order to determine how many curry levels we need. We also used Ramda’s always to create a function that regardless of the arguments passed will always return target.

Finally, the validate() function is completely agnostic about the arbitrary validations. It became a generic validator which will work for any number of validation functions returning a Validation subtype, so we could even rename the password argument to just target.

There is still room for improvement though. Let’s revisit our predicate based validations and try to remove the duplication in there.

  
// ...

const isPasswordLongEnough = password =>
  password.length > 6 ?
    Success(password) :
    Failure(["Password must have more than 6 characters"]);

const isPasswordStrongEnough = password =>
  /[\W]/.test(password) ?
    Success(password) :
    Failure(["Password must contain special characters"]);

It’s clear that those two functions have the exact same logic except by (a) the predicate being executed and (b) the associated error message. Let’s extract this duplicated logic and try to push it to the validate() function.

  
// ...

const accumulateValidationResults = target => (acc, validation) =>
  acc.ap(validation.predicate(target) ? Success(target) : Failure([validation.error]));

const validate = (validations, target) =>
  reduce(
    accumulateValidationResults(target),
    Success(curryN(length(validations), always(target))),
    validations);

const passwordValidations = [
  {
    error: "Password must have more than 6 characters",
    predicate: password => password.length > 6
  },
  {
    error: "Password must contain special characters",
    predicate: password => /[\W]/.test(password)
  }
];

validate(passwordValidations, "foo");
// => Validation.Failure([
//      "Password must have more than 6 characters.",
//      "Password must contain special characters."
//    ])

And that was the last step of our journey. The validate() function is now able to determine validation results based only on a data structure representing arbitrary validations to be performed against a target object.

Success / Failure

Why is it useful to return a container as a result?

Well, for at least two reasons:

Success and Failure have explicit semantics about what happened and can be used to drive subsequent actions without leaving the current thread.
Those containers provide a useful interface to safely manipulate the results in a monadic way. You can, for instance, define callbacks for success and failure and fold validate() return value accordingly.

  
// ...

const passwordValidations = [
  {
    error: "Password must have more than 6 characters",
    predicate: password => password.length > 6
  },
  {
    error: "Password must contain special characters",
    predicate: password => /[\W]/.test(password)
  }
];

const onSuccess = password => "This is an excellent password!";
const onFailure = errors => "This is invalid because:\n" + errors.map(e => "* " + e).join("\n");
 
const message = validate(passwordValidations, "foo").fold(onFailure, onSuccess);
 
console.log(message);
// => This is invalid because: 
// => * Password must have more than 6 characters. 
// => * Password must contain special characters.

Predicado

The above exercise on defining a Folktale based validate() utility resulted in a tiny but brave library called Predicado. Check it out and feel free to send feedback.

Maybe Better Reducers

2016-09-20T22:48:00+00:00

Redux is one of them most elegant ideas in the Javascript ecosystem in the last few years. Inspired by Facebook’s Flux architecture, this small but brave library helps you to build applications based primarily on immutable data structures and pure functions. Folks with bias on Functional Programming (like me) will feel at home with Redux.

In summary, a Redux app is a container for a immutable data structure representing the current state. That container accepts actions which are just plain objects describing user intent. Once an action is accepted, a reducer function runs and creates the next state based on the action and the current state. The reducer function then is the core of a Redux app.

The Reducer

Redux doesn’t impose any restrictions on the way you define your reducer, as long as it comforms with the following signature:

reducer :: (State, Action) -> State

Most examples all over the internet - including the one in the official documentation - will suggest you to define the reducer function using a switch statement on the action type.

function reducer (state = [], action) {
  switch (action.type) {
    case "ADD_TODO":
      return state.concat({
        text: action.text,
        completed: false
      });
    case "TOGGLE_TODO":
      return state.map((todo, index) =>
        index === action.index ?
          Object.assign({}, todo, { completed: !todo.completed }): todo
      );
    default:
      return state;
  }
}

This is literally the reducer suggested in the Redux official docs. I wonder if people are really following this approach (I hope not). As you can see, the code above is very busy to read and not at all modular. Hopefully, we can do better than that. Let’s see.

Introducing Maybe

The algorithm of the reducer we just saw can be described as:

It chooses a behavior by pattern-matching the action.type;
Then, it executes the matching behavior returning a new state;
In case there is no match, it returns the unchanged state.

Turns out that logic can be accuratelly implemented using a Maybe container, as the Haskell programmers would probably suggest here. Java 8 developers can think of it as an Optional. In Javascript, we can get Maybe from the amazing Folktale lib. Let’s see the result.

import Maybe from "data.maybe";

const reducer = (state, action) => 
  Maybe
    .fromNullable(actionHandlers[action.type])
    .map(handler => handler(state, action))
    .getOrElse(state);

The Maybe version of our reducer relies on an addional step: we extracted all the action-specific behaviors to separate functions, which are accessible in a map we called actionHandlers.

const ADD_TODO = (state, action) =>
  state.concat({ text: action.text, completed: false });

const TOGGLE_TODO = (state, action) =>
  state.map((todo, index) =>
    index === action.index ?
      Object.assign({}, todo, { completed: !todo.completed }): todo);

const actionHandlers = { ADD_TODO, TOGGLE_TODO };

Now each action handler is an independent function with a single and succint responsibility. The same applies to the reducer which uses the Maybe container to drive the logic around the pattern matching. And just that.

Those who are not familiar with Maybe can think of it as a container for a value that might or might not be present. In our example, an action type might or might not be found in the map. The .fromNullable constructor does that for us: it gets an actual function or it gets nothing in case there is no match. Next, we call .map which executes a present behavior. The interesting thing about .map is that is does not run in case the Maybe is holding an empty value. The last call .getOrElse returns the value from inside the container if the value is present. Otherwise, it uses the default argument, which in our case is the unchanged state.

Conclusion

Redux is great not only because it is so simple and concise but also because it is based on strong Functional Programming concepts. Additionally, Maybes, Eithers, Validations and other containers from Folktale are very handy in a diversity of situations, but specially useful when it comes to functional error handling. The reducer example is just a small demonstration on how well the Redux idea fits with other FP based libs. For a complete and more detailed example take a look at my Redux + Folktale + Ramda Todo App.

Update

The idea behind this post inspired the development of the Redux Definer NPM package. Pretty handy to avoid reducer boilerplate code. In case you also want to avoid action creators boilerplate, you can go straight to Redux Actions.

The Future is Functional

2016-09-15T01:48:00+00:00

This week my friend @ffafara and I joined a meetup organized by the Philadelphia Java Users Group. There was a presentation about some interesting details on Java SE 8 and also some highlights on what’s coming up in version 9. The discussion was quite nice, specially regarding the Functional Programming features like lambdas, functional interfaces, streams, and optionals.

One thing called our attention though: the presenter’s opinion about the relevance of FP concepts in the future of mainstream programming. Although he highlighted the potential benefits of those concepts in the context of Java, he does not believe FP could be a replacement for the Object Oriented mindset when it comes to general application development.

This point of view, though, is not at all a rare one. It’s quite frequent to observe people being averse to FP (usually, influenced by old-fashioned software architects), not only due to the intellectual pain of the paradigm shift, but mainly because they still believe OO should be the primary tool for building systems.

Well, it seems they are missing an important point here.

What they probably don’t yet realize is that OO itself is the fundamental reason behind most of the pain points in their daily programming routine:

Null pointers every now and then;
Unpredictable side effects;
Artificial data abstractions obscuring the domain concepts;
Layers, and more layers of mutable state;
Flaky unreliable tests;
You name the next one…

In a world every day more distributed and asynchronous those inconveniences are gradually becoming more apparent and frequent. The reason is simple: OO’s basic assumption is mutable state. And mutable state doesn’t deal well with the increasingly simultaneity of modern computer systems. Let me call Uncle Bob to explain that to us:

You are almost certainly reading this on the screen of your computer. How many cores does it have? I’m typing this article on a MacBook Pro with 4 real cores. My previous laptop had two cores. And the one before that had just one. My next laptop will truly have eight cores; and the one after that will likely have 16. How are you going to take advantage of every computer cycle available to you when your computer has 4096 cores in it?

FP Basics E1: What’s Functional Programming all about? by Uncle Bob.

Honestly, we programmers can barely get two Java threads to cooperate. For over half a century programmers have made the observation that the processes running in a computer are concurrent, not simultaneous. Well, boys and girls, welcome to the wonderful world of simultaneity! How are you going to deal with it?

Pure functions impose us two clear constraints: explicit return values and no side effects. These constraints are important enablers for thread safety and predictable executions. When there are no true variables, there is nothing harmfully unexpected one thread can cause to another. Therefore, parallelizing computation becomes trivial. Besides that, when data is immutable and behavior is idempotent, your code becomes easier to test and debug, since people are no longer allowed to sneak in their nasty side effect based workarounds everywhere.

Clearly, if the value of a memory location, once initialized, does not change during the course of a program execution, then there’s nothing for the 131072 processors to compete over. So that’s the big deal about functional languages; and it is one big fricking deal.

Of course OO is still a good choice for most single threaded applications. However, in a world every day more and more event-driven, asynchronous, and distributed, the insistence on the OO supremacy has been leading to fragile systems which are hard to test, debug, and extend. FP, on the other hand, fits perfectly the requirements of that scenario. That’s why we’ve been observing an increasing focus on FP languages, libraries, and tools by big tech companies like Facebook for instance. Although FP might be challenging to learn in the beginning, the passionate programmer most definitely finds it a fascinating and beautiful concept, which is, above everything else, powerful.

I’ll leave you with Uncle Bob again:

Robert C. Martin - Functional Programming: What? Why? When? from NDC Conferences on Vimeo.

The Developers Conference

2016-07-19T11:00:00+00:00

This month I attended The Developer’s Conference in Sao Paulo together with my ThoughtWorks friends from Brazil. There, I presented three different sessions in three different tracks: Microservices, Startups, and Agile.

Oi pessoas que estão no @TheDevConf! #TWnoTheDevConf #ThoughtWorks pic.twitter.com/zuLXeLYi6K
— ThoughtWorks Brasil (@thoughtworksbr) July 6, 2016

Event Sourcing & Microservices

Microservices are becoming mainstream in the last few years, specially after Sam Newman’s book. When it comes to both building new systems from scratch or decomposing legacy monoliths in smaller pieces, this architectural style gives you interesting advantages like deployment flexibility, technology heterogeneity, and governance decentralization.

However, in the traditional microservices approach based on REST over HTTP, there are challenges like performance, data consistency, distributed transaction management, and also the limitations of the HTTP protocol methods to represent user intent.

With Event Sourcing you describe the application data as a sequence of immutable domain events. Those events are caused by the execution of custom commands which represent user intent. In this scenario, microservices communication happens asynchronously by the broadcasting and listening of events. That style gives you important architectural advantages like auditing capability, historical analysis, and global consistency. These features are specially useful in domains like banking, for instance.

Based on recent experiences in the dense U.S. banking domain, I presented to the public a short summary on how Event Sourcing combined with Microservices was a successful solution to my team. The presentation included basic concepts as well as a description of technical aspects like CQRS, infrastructure, and technologies. Here are the slides (Portuguese only):

Interviewing Developers

It would be ludicrous to think of hiring a juggler without first seeing him perform.

DeMarco & Lister, Peopleware.

When it comes to recruit developers, the traditional approach with paper resumes, oral interviews, and whiteboard puzzles is seldom good enough to determine whether the candidate is a match or not. Instead, it is often more useful to put candidates to exercise their skills in a close to real world scenario.

For that to work, we need to setup an adequate environment and perform activities which would give the candidate a chance to experience realistic situations. Even though sometimes this is something hard to go about, there are some strategies you can adopt in order to simulate a good enough setup for technical interviews.

In the last decade, I’ve been interviewing a lot of candidates for ThoughtWorks and I’ve watched our recruiting process evolution closely. In this talk, I shared a short summary of that experience to the public, including highlights of what the relevant literature says, targeting technical interviews for developers. Here are the slides (Portuguese only):

Interesting fact is that this is the fourth time we speak about this topic publicly, which is a sign of interest from the public. The other appearances are:

The Developers Conference 2015 - Florianopolis, Brazil
Agile Brazil 2015 - Porto de Galinhas, Brazil
Tecnologicamente Arretado Podcast

Feature Leads

Have you - developer - ever felt your tech lead becoming a decision proxy or a information bottleneck? Have you - tech lead - ever felt overwhelmed with the amount of management activities and context switching? Feature Leads can be a good alternative for you, specially if you are in medium to large team with a medium to large backlog.

This was the topic of my last talk in the last day of the conference, in the Agile track. Based on the original Feature Lead/Champion idea described by Pat Kua and Ryan Oglesby, I described to the audience some recent experiences some of the teams I’ve been working with had with this technique. Check out the slides (in Portuguese):

Videos of all the conference talks can be found here. They aren’t free though :(

The Beauty of Recursion

2016-03-31T11:00:00+00:00

Recursion is one of the most fascinating and elegant concepts in Computer Science. The feeling of figuring out recursive algorithms & data structures is a special one for a software developer. In this context, there is probably no better source than The Little Schemer book for both understanding and satisfying your recursive appetite.

Although this might seen like just a book about Scheme, I’d say this is mostly a book about thinking recursively. Based entirely on questions, puzzles, and exercises which guide you gradually from the most basic concepts, like atoms and s-expressions, to the most mind-blowing ones, like the Y Combinator, the teaching method is quite engaging and very unusual when compared to traditional programming books.

The Little Schemer seems to be a popular book among Clojure developers like my friend Marcos. Even though there are idiomatic differences to Scheme, it is totally possible to implement all the exercises in the book using Clojure. Here is my test-driven version: github.com/vvgomes/little-schemer

The Y Combinator

By far, the most difficult challenge in the book is the Y Combinator. In summary, the goal is to describe recursion formally using Lambda Calculus theory, which in practice means pretty much defining recursive functions with anonymous functions only. In the example given in the book we move from a traditional recursive function definition like this:

(def length
  (fn [l]
   (cond
     (null? l) 0
     :else (add1 (length (cdr l))))))

… to a Y Combinator based definition, like the following:

(def Y
  (fn [le]
    ((fn [f] (f f))
      (fn [f] (le (fn [x] ((f f) x)))))))

(def length
  (Y
    (fn [length]
      (fn [l]
        (cond
          (null? l) 0
          :else (add1 (length (cdr l))))))))

There are around 17 steps involved to get from the initial to the final stage (you can see all the steps here) and you usually don’t get it in the first time. Not even in the second one. However, if after the third time you still don’t get it, I’d recommend you to take a look at this very good explanation by Jim Weirich and try again. That’s what I did.

Does your hat still fit?

Perhaps not after such a mind stretcher.

Better Spring Data Repositories

2016-03-18T11:00:00+00:00

So, you find yourself having to make changes in a Java application where folks are using Spring Data based repositories. Everything looks good in the beginning, no need to implement any ORM code from scratch since Spring Data will figure that out for us. Let’s see an example.

@Entity
public class Customer {
  @Id
  private Integer id;
  private String firstName;
  private String lastName;
  private LocalDate birthDate;
  private LocalDate enrollment;
  @Enumerated(STRING)
  private FinancialStatus status;

  // more stuff
}

public enum FinancialStatus {
  GOOD_STANDING,
  DEBT,
  BANKRUPT
}

Given the above Customer entity, imagine you have the following repository interface:

public interface CustomerRepository extends CrudRepository<Customer, Integer> {

  List<Customer> findByLastNameAndStatusIn(String lastName, List<FinancialStatus> statuses);

  List<Customer> findByFirstNameAndLastNameOrderByEnrollmentDesc(String firstName, String lastName);
}

As you may have realized, just by extending CrudRepository you get all the basic stuff like save, findAll, findOne for free. Then, anything more complex, like the two declared methods, will be interpreted by Spring Data in runtime based on name conventions.

The Problem

As time goes by and business starts to ask more complicated questions, your repository starts to get weird. The following code is an example that illustrates a real world situation I’ve faced recently:

public interface CustomerRepository extends CrudRepository<Customer, Integer> {

  List<Customer> findByLastNameAndStatusInOrderByEnrollmentDesc(String lastName, List<FinancialStatus> statuses);

  List<Customer> findByFirstNameAndLastNameAndBirthDateIsLessThanAndStatusInOrderByEnrollmentDesc(String firstName, String lastName, LocalDate birthDateLimit, List<FinancialStatus> statuses);

  List<Customer> findByFirstNameAndLastNameAndBirthDateIsBetweenAndStatusInOrderByEnrollmentDesc(String firstName, String lastName, LocalDate birthDateFrom, LocalDate birthDateTo, List<FinancialStatus> statuses);
 }

At this point, the maintenance of this particular repository starts to become problematic for obvious reasons:

Method names are too long and excessively verbose;
They’re not easy to read;
It’s not straightforward to build new queries (as we need to rely on Spring Data ability to understand the method name);
Some method names don’t even result in a valid English sentence.

Imagine yourself talking to your pair:

Hey Akon, I think we could use the find by first name and last name and birth date is between and status in order by enrollment desc method here.

Awkward, to say the least.

Another problem that may call your attention is the duplication of criteria among the query methods: most of them are repeating the same filters over and over again. And as the amount of filters increases, the argument list also increases making it more difficult to figure out what is what.

Wouldn’t that be great if we could extract those filters in smaller reusable pieces?

JPA Specifications

With the Spring Data JPA Specifications API we can implement this idea of extracting all filter and sorting criteria in independent pieces. Let’s see how the new version of the the repository looks like:

public interface CustomerRepository extends CrudRepository<Customer, Integer>, JpaSpecificationExecutor<Customer> {
  class Specifications {
    public static Specification<Customer> firstNameIs(String firstName) {
      return (root, query, cb) -> cb.like(root.get("firstName"), firstName);
    }

    public static Specification<Customer> lastNameIs(String lastName) {
      return (root, query, cb) -> cb.like(root.get("lastName"), lastName);
    }

    public static Specification<Customer> statusIn(FinancialStatus... statuses) {
      return (root, query, cb) -> cb.isTrue(root.get("status").in(statuses));
    }

    public static Specification<Customer> birthDateIsBetween(LocalDate from, LocalDate to) {
      return (root, query, cb) -> cb.between(root.get("birthDate"), from, to);
    }

    public static Specification<Customer> birthDateIsBefore(LocalDate limit) {
      return (root, query, cb) -> cb.lessThan(root.get("birthDate"), limit);
    }

    public Sort orderByEnrollmentDesc() {
      return new Sort(Sort.Direction.DESC, "enrollment");
    }
  }

As you can see in this new version we had to extend JpaSpecificationExecutor. That makes it possible for our repository to accept Specification based queries using the following method:

List<T> findAll(Specification<T> spec, Sort sort);

After that, we’ve implemented each of the necessary filters and sorting as static methods and got rid of all the old long-named methods. The inner Specifications class works as a namespace just to avoid Spring Data trying to interpret our filters as regular CrudRepository methods.

Without getting too deep in details about root, query, cb (criteria builder), what our new methods do is to return reusable specifications. And that should be enough to allow us to replace calls to

findByFirstNameAndLastNameAndBirthDateIsLessThanAndStatusInOrderByEnrollmentDesc

with the following code:

List<Customer> results = repository.findAll(
   where(firstNameIs("John"))
     .and(lastNameIs("Doe"))
     .and(birthDateIsBefore(eighteenYearsAgo))
     .and(statusIn(GOOD_STANDING, DEBT)),
   orderByEnrollmentDesc());

The example above, alone, would be enough to make most people happy when it comes to readability and flexibility. Still, we could even go further and define default methods with meaningful names in our new repository using our new specifications like this:

  default List<Customer> inDebtWithLastName(String lastName) {
    return findAll(where(statusIn(DEBT)).and(lastNameIs(lastName)));
  }

Conclusion

The magic Spring Data does, allowing us to define queries only by writing method names is definitely cool for most of the simpler situations. However, it doesn’t scale well as queries become more complex. The Specification API, as described, seems to be a good alternative. Keep that in your tool-belt.