How to solve it? Complexity in code - Improving flow control through functional pipelines

Code: https://gitlab.com/danbunea/improving-control-flow-in-code-using-functional-pipelines

Why?

You:

Why?

Me:

Because code is better...

You:

Code is better? How?

Me:

Because is it easier to: 

• read
• extend
• debug

You:

Prove it!

Me:

Ok let's look at the following problem:

In practice

We have to write an endpoint which returns an offer by its id:

GET /offer-by-id/:offer-id
Possible results:

- 200 {"offer-id":2, "offer-data":""}
- 400 {"errors":["The id you provided is invalid"}]}
- 404 {"errors":["The id you provided cannot be found"}]}
IMPORTANT: if an id is valid but not found the deposit must be notified!

Solution 1: IFs

The problem could be solved by:

Or in code:

Whenever we have ifs in code, it becomes dificult to read so maybe we could simplify it, making it more like:

step 1
then step 2
then step 3

Or more like

validate
then find offer
then jsonify

Hmm, can we?

Solution 2: Exceptions

We'll use exceptions to break out flow:



The solution above is very present in Java, even if the way the errors are caught might not be this explicit.

Because it's a fairly simple example we could also use the strategy pattern. Basically we use ifs to choose a strategy, then we execute it to give us the result. 

Solution 3: strategy pattern

Functional programming gives us sever possibilities using pipes. 

What is a pipe?

A pipe makes sure that steps get executed in a certain order and that the result of a step is passed to the next step. Like:

In our case we'd like something like:

validate(requestId)
.then(findOfferById)
.then(jsonify)

Solution 4 Functional either or railway oriented programming

This is a very common solution in typed functional languages such as F# or Haskel, but it's becoming very common in Java as well.

Having a single pipeline is very beautiful but how do we handle errors? By using two parallel pipes. A pipe for the happy path and a path for the errors. Basically all our functions can return either a SuccesfullResponse or a ErrorResponse. This will get passed on to the next function that will process it and return again either a SuccesfullResponse or a ErrorResponse.

In F# we'd have something like:

findOrderById: IResponse -> SuccesfullResponse | ErrorResponse

we might describe it like:

SuccesfullResponse | ErrorResponse aFunction(SuccesfullResponse | ErrorResponse response)

For the happy path we'd have:

And if we get a validation error in the first step we'll have:

And the code, will have a class for Success and a class for Error. Each will inherit an IResponse and will implement two functions receiving a function (lambda) then and fail. In Succes we'll return the result of applying the function on what we have on then and the data we have on fail, and on Error we'll do exactly the opposite. 

Imaging you'd replace then with map and fail with orElseGet, doesn't that sound like an optional?



In dynamic languages, there are other options. But first let's describe functional composition. 

Functional composition is when you combine 2 (or more) functions into one, then apply it. It's pretty much like pipe but you may do the composition at runtime. 

Now we can look at the two options. First is:

Solution 5: pipeline with flag

We'll pass through the pipe an object that contains a flag which tells you whether there were errors before. Basically using a value in your data instead of using the type of the data (has response property instead of type: Successful or Error).

In our case, the flag is whether a response has been set already:

if(state.response) return state;

Solution 6: pipeline + overflow pipeline 

The second option is to have a pipeline and an overflow pipeline.  

 We'll use exceptions once again to bypass the normal pipeline and go to the overflow pipeline.

Unlike the previous two options when an exception happend it will jump straight to the end, bypassing the next steps directly.



The code: https://gitlab.com/danbunea/improving-control-flow-in-code-using-functional-pipelines/tree/stage-1

Readability

In terms of readability it could be a lot easier, to see the code as a pipeline:

step 1
then step 2
then step 3

While also handling the errors:

step 1
then step 2
then step 3
fail on-error

or 

safe(
   step 1
   then step 2
   then step 3
)

Extensibility

But what about extensibility?

We now have to modify our endpoint

1. to check if the offers are still active. If the aren't we need to return an error
2. to update the number of times the offer has been accessed

GET /offer-by-id/:offer-id

Possible results:

- 200 {"offer-id":2, "offer-data":"", "active":true, "requests":1}
- 400 {"errors":["The id you provided is invalid"}]}
- 400 {"errors":["The offer expired"}]}
- 404 {"errors":["The id you provided cannot be found"}]}

IMPORTANT: if an id is valid but the offer expired the deposit must be notified!

We changed the tests:

Then we change the code and we can look at how
For solution 1:

The way we solved it was to add more ifs inside an existing if thus increasing the cyclomatic complexity of the solution making it even harder to read. And real life code tends to be more complex than this.


For solution 2:

We did:

• added a new exception
• changed the code inside the try catch block
• changed the code in the catch

For solution 4:

What did we do:

• added two new functions, completely independent 
• modified an existing one
• added steps to the pipeline

For solution 5:

What did we do:

• added a new function
• heavily modified an existing one
• added a step to the pipeline

For solution 6:

What did we do:

• added a two new functions
• added them as steps to the pipeline

No existing code modified! (except for the pipeline, which is expected)

The code: https://gitlab.com/danbunea/improving-control-flow-in-code-using-functional-pipelines/tree/stage-2

The merge request in gitlab: https://gitlab.com/danbunea/improving-control-flow-in-code-using-functional-pipelines/merge_requests/3/diffs#faff669626dfa73714964353a02a5101dce1b3a7

Debugging

Let's say we need to start logging what is going on. For ifs we may end up with:

Not exactly easy to know where to insert the logging. 
But for a pipeline, we just need to insert some tracing function between the steps:

giving us:

or

And the log is like:

Last, let's see where pipelines could be used.

Frontend/Javascript with Promises:

or front end Clojurescript:

or backend Python:

or Java pipelines

or

The example comes from the book "Functional Style" by our colleague Richard Wild  https://functional.works-hub.com/learn/the-functional-style-part-5-higher-order-functions-i-function-composition-and-the-monad-pattern-bc74a?utm_source=blog&utm_medium=blog&utm_campaign=j.kaplan

Conclusion

Typed languages (including Java) you should use Either.

Dynamic languages the best would be pipeline with exceptions.

Simple , robust code: part one, simplicity

1. Simple as the oposite of complex

Complexity in software is the root of all evil, and simplicity is the oposite of complexity. Simple is not the same as easy, because sometimes we make software complex just because it is easy (think of adding a library from which you need just a function, which then needs to be upgraded and it's incompatible with other libraries etc).

A complex sistem is like this, where is is very hard to figure out what is going on, thus it cannot be debugged, extended or changed:

and a simple one is the oposite:

2. Simplicity in software DATA and FUNCTIONS*

We use computers to compute (apply functions) some data we need (the final state of a system), given some initial data (initial state of the system). So if we drastically reduce what software does, we end up with just data and functions.

Example:

Obviously this sounds overly simplistic, real code is more complex, more functions are needed.

function greed(name){
var a = ["hello ", name];
var b =capitalize_first_letter(a);
var c =concat(b);
return c;
}


or we could:

Which starts to look like a pipe, where you send the initial_state, and expect at the end the final state.

Now if we need to solve a real world problem, I guess we could solve it by having:
- lots of simple functions, that take as input one parameter and return one parameter
- because the have one parameter in and one parameter our they can be composed
- simple functions put together as a pipeline and can solve very complex problems in a very simple way

3. Functional composition

Now we could compose the two functions into just one:

4. Example: From complex to simple using functional composition

A few years ago, I made a practical example. I'll add it simplified here.

Requirement: in the json that we receive on a server, we need to have a key “measurement”, that is mandatory, cannot be null, needs to be a string and cannot be empty string, Then we also need to make sure the length of the string is between 3 and 8 characters, and cannot be some reserved words like “password” or “archived". So the code is like:

 def validate_simplest_json(json):  
   errors = []  
   if not json.has_key("measurement"):  
     errors.append("measurement cannot be missing")  
   else:  
     if json["measurement"]==None:  
       errors.append("measurement cannot be null")  
     else:  
       if not isinstance(json["measurement"], str) and not isinstance(json["measurement"], unicode):  
         errors.append("measurement needs to string or unicode")  
       else:  
         lenm=len(json["measurement"].strip())  
         if lenm==0:  
           errors.append("measurement cannot be an empty string")  
         else:  
           if lenm<3: data-blogger-escaped-div="">  
             errors.append("measurement needs at least 3 characters")  
           elif lenm&gt;10:  
             errors.append("measurement needs at most 10 characters")  
           elif json["measurement"].strip().lower() in ["archived","password"]:  
             errors.append("measurement has a value which is not allowed")  
   return errors  

Removing complexity can mean, more linear code, and an initial state, and simple composable functions:

 ValidationState = namedtuple("ValidationState","json key errors exit”)  

then I will extract the actual validations in simple functions, like:

 def validate_simplest_json_imperative_linear_with_state(json):  
   initial_state = ValidationState(json=json, key="measurement",errors=[], exit=False)  
   
   state = validate_key_exists(initial_state)  
   
   if not state.exit:  
     state = validate_not_null(state)  
   
   if not state.exit:  
     state = validate_string_or_unicode(state)  
   
   if not state.exit:  
     state = validate_not_empty_string(state)  
   
   if not state.exit:  
     state = validate_length(state, 3,10)  
   
   if not state.exit:  
     state = validate_not_in(state, ["archived","password"])  
   
   return state.errors  

And the functions are like:

 def validate_key_exists(state):  
   print validate_key_exists.__name__,state  
   if not key_exists(state.json,state.key):  
     return state._replace(errors = state.errors+["{0} cannot be missing".format(state.key)])._replace(exit=True)  
   return state  
   
 def validate_not_null(state):  
   print validate_not_null.__name__,state  
   if value_null(state.json,state.key):  
     return state._replace(errors = state.errors+["{0} cannot be null".format(state.key)])._replace(exit=True)  
   return state  
   

...

The code looks is now a series of functions that run with the result of the previous function if the exit parameter is not set to True. So basically having 2 functions f,g they’ll be composed like:

initial_state = …

state = f(initial_state)

if not state.exit:

    return g(state)

And putting this in a function:

 def compose2(f, g):  
   def run(x):  
     result_f = f(x)  
     if not result_f.exit:  
       return g(result_f)  
     else:  
       return result_f  
   return run  

 #compose n functions  
 def compose(*functions):  
   return reduce(compose2, functions)  

And now the final validation code:

 def validate_simplest_functional_composition(json):  
   initial_state = ValidationState(json=json, key="measurement",errors=[], exit=False)  
   
   composed_function = compose(

          validate_key_exists, 

          validate_not_null,

          validate_string_or_unicode,

          validate_not_empty_string, 

          create_validate_length(3, 10), 

          create_validate_not_in(["archived","password"]))  

   final_state = composed_function(initial_state)  
   
   return final_state.errors  

It is much better. It basically says: having an initial start of the system, run all these functions (validators) and at the end get a final state. Code: http://runnable.com/VNMhoTKLSn9Tm0GI/fighting-complexity-through-functional-composition-for-python

And it is:

5. So where can I use this?

If you're a backend developer, you can use it on a server (python example):

@mod.route('/api/1/save/', methods=['POST'])
@pi_service()
def generic_save(version=1, typ=None):
    composed_func = compose_list(
    [
        can_write("tags"),
        change("json", request.json),
        change("session", get_session()),
        change("type", get_pi_type(typ)),
        change("object", None),
        change("transformer", get_pi_transformer(typ)),
        get_database_object,
        transform_from_json,
        save_database_object,
        index_tag_or_tag_group,
        pi_transform_to_json,
   ])
return composed_func({})

or in a PDF generating server, written in Clojure over Apache Batik (using transducers but that's another discussion)

You could use javascript promises for piping, with React, if you're a front-end developer. The state of the system the model (immutable) and rendering is done views.render:

StoryboardController.prototype.move_point_by = function(page_object, point_index, dx, dy) {
    pi.startWith(model,"MOVE POINT BY")
        .then(function move_point_by(state){
            pi.info("move point by", page_object, point_index, dx, dy);
            var cursor = get_selected_layer_cursor(state) + ".children" + find_cursor_pageobject(page_object, state);
            if (cursor) {
                var point_cursor = cursor+".points["+point_index+"]";
                var point = pi.pi_value(state, point_cursor);
                var changes = {};
                var nx=point.x+dx;
                var ny=point.y+dy;
                changes[point_cursor+".x"]=nx;
                changes[point_cursor+".y"]=ny;

                state = pi.pi_change_multi(state, changes);

                return resize_shape(state, cursor);
            }
            return state;
        })
        .then(views.render)
        .then(swap_model)
        .then(REST.try_save_page)
}

or

or in Clojurescript, where the state of the system is an atom (model) and every time it changes, the view is rerendered:

6. Conclusion

Using this model, code is easier to understand, debug, change, extend. Why: 

- all the data is in a place

initial_state = ValidationState(json=json, key="measurement",errors=[], exit=False) 

- functions are simple 

 def validate_key_exists(state):  
   print validate_key_exists.__name__,state  
   if not key_exists(state.json,state.key):  
     return state._replace(errors = state.errors+["{0} cannot be missing".format(state.key)])._replace(exit=True)  
   return state 

- intermediary states can be easily debugged

 composed_function = compose(

          validate_key_exists, 

          validate_not_null,

          debug,

          validate_string_or_unicode,

          validate_not_empty_string, 

          create_validate_length(3, 10), 

          create_validate_not_in(["archived","password"]))  

   final_state = composed_function(initial_state)  

 def debug(state):  
   print state.json, state.key, state.errors, state.exit
   return state 

- data changes flow in a single direction

In part two: robustness, we'll see how we could also make the code robust, by making the code run transactionally same as databases: either all runs or none and the state gets reverted to the previous one.