Friday, July 28, 2023

Datomic as-of and since, in practice

 Considering we have a simple Datomic database, where we do 3 transactions:

First we add an app. Then we add a chat "c1" in that app. Then we add another chat "c2" in the same app.

Looking at the transaction history:





Now we can pull, at the different times of the transactions, using as-of meaning we'll only see the results as they were at that time:


But another interesting thing, is seeing the changes since a particular time using since (think you're asking only for changes since a particular time):





Saturday, September 21, 2019

How to solve it? Complexity in code - Improving flow control through functional pipelines


Code: https://gitlab.com/danbunea/improving-control-flow-in-code-using-functional-pipelines

Why?


You:

Why?

Me:

Because code is better...

You:

Code is better? How?

Me:

Because is it easier to: 
  • read
  • extend
  • debug
You:

Prove it!

Me:

Ok let's look at the following problem:

In practice


We have to write an endpoint which returns an offer by its id:

GET /offer-by-id/:offer-id

Possible results:

- 200 {"offer-id":2, "offer-data":""}
- 400 {"errors":["The id you provided is invalid"}]}
- 404 {"errors":["The id you provided cannot be found"}]}

IMPORTANT: if an id is valid but not found the deposit must be notified!

Solution 1: IFs

The problem could be solved by:



Or in code:


Whenever we have ifs in code, it becomes dificult to read so maybe we could simplify it, making it more like:

step 1
then step 2
then step 3

Or more like

validate
then find offer
then jsonify

Hmm, can we?


Solution 2: Exceptions


We'll use exceptions to break out flow:



The solution above is very present in Java, even if the way the errors are caught might not be this explicit.

Because it's a fairly simple example we could also use the strategy pattern. Basically we use ifs to choose a strategy, then we execute it to give us the result. 


Solution 3: strategy pattern





Functional programming gives us sever possibilities using pipes. 

What is a pipe?

A pipe makes sure that steps get executed in a certain order and that the result of a step is passed to the next step. Like:




In our case we'd like something like:

validate(requestId)
.then(findOfferById)
.then(jsonify)

Solution 4 Functional either or railway oriented programming

This is a very common solution in typed functional languages such as F# or Haskel, but it's becoming very common in Java as well.

Having a single pipeline is very beautiful but how do we handle errors? By using two parallel pipes. A pipe for the happy path and a path for the errors. Basically all our functions can return either a SuccesfullResponse or a ErrorResponse. This will get passed on to the next function that will process it and return again either a SuccesfullResponse or a ErrorResponse.

In F# we'd have something like:

findOrderById: IResponse -> SuccesfullResponse | ErrorResponse

we might describe it like:

SuccesfullResponse | ErrorResponse aFunction(SuccesfullResponse | ErrorResponse response)

For the happy path we'd have:




And if we get a validation error in the first step we'll have:




And the code, will have a class for Success and a class for Error. Each will inherit an IResponse and will implement two functions receiving a function (lambda) then and fail. In Succes we'll return the result of applying the function on what we have on then and the data we have on fail, and on Error we'll do exactly the opposite. 

Imaging you'd replace then with map and fail with orElseGet, doesn't that sound like an optional?



In dynamic languages, there are other options. But first let's describe functional composition. 

Functional composition is when you combine 2 (or more) functions into one, then apply it. It's pretty much like pipe but you may do the composition at runtime. 

Now we can look at the two options. First is:


Solution 5: pipeline with flag


We'll pass through the pipe an object that contains a flag which tells you whether there were errors before. Basically using a value in your data instead of using the type of the data (has response property instead of type: Successful or Error).


In our case, the flag is whether a response has been set already:

if(state.response) return state;



Solution 6: pipeline + overflow pipeline 

The second option is to have a pipeline and an overflow pipeline.  



 We'll use exceptions once again to bypass the normal pipeline and go to the overflow pipeline.




Unlike the previous two options when an exception happend it will jump straight to the end, bypassing the next steps directly.



The code: https://gitlab.com/danbunea/improving-control-flow-in-code-using-functional-pipelines/tree/stage-1

Readability


In terms of readability it could be a lot easier, to see the code as a pipeline:

step 1
then step 2
then step 3

While also handling the errors:

step 1
then step 2
then step 3
fail on-error

or 

safe(
   step 1
   then step 2
   then step 3
)




Extensibility


But what about extensibility?

We now have to modify our endpoint

  1. to check if the offers are still active. If the aren't we need to return an error
  2. to update the number of times the offer has been accessed


GET /offer-by-id/:offer-id

Possible results:

- 200 {"offer-id":2, "offer-data":"", "active":true, "requests":1}
- 400 {"errors":["The id you provided is invalid"}]}

- 400 {"errors":["The offer expired"}]}
- 404 {"errors":["The id you provided cannot be found"}]}


IMPORTANT: if an id is valid but the offer expired the deposit must be notified!

We changed the tests:



Then we change the code and we can look at how
For solution 1:



The way we solved it was to add more ifs inside an existing if thus increasing the cyclomatic complexity of the solution making it even harder to read. And real life code tends to be more complex than this.


For solution 2:





We did:

  • added a new exception
  • changed the code inside the try catch block
  • changed the code in the catch


For solution 4:



What did we do:

  • added two new functions, completely independent 
  • modified an existing one
  • added steps to the pipeline


For solution 5:


What did we do:

  • added a new function
  • heavily modified an existing one
  • added a step to the pipeline


For solution 6:



What did we do:

  • added a two new functions
  • added them as steps to the pipeline
No existing code modified! (except for the pipeline, which is expected)


The code: https://gitlab.com/danbunea/improving-control-flow-in-code-using-functional-pipelines/tree/stage-2

The merge request in gitlab: https://gitlab.com/danbunea/improving-control-flow-in-code-using-functional-pipelines/merge_requests/3/diffs#faff669626dfa73714964353a02a5101dce1b3a7

Debugging


Let's say we need to start logging what is going on. For ifs we may end up with:



Not exactly easy to know where to insert the logging. 
But for a pipeline, we just need to insert some tracing function between the steps:



giving us:




or


And the log is like:




Last, let's see where pipelines could be used.

Frontend/Javascript with Promises:





or front end Clojurescript:




or backend Python:



or Java pipelines



or





The example comes from the book "Functional Style" by our colleague Richard Wild  https://functional.works-hub.com/learn/the-functional-style-part-5-higher-order-functions-i-function-composition-and-the-monad-pattern-bc74a?utm_source=blog&utm_medium=blog&utm_campaign=j.kaplan

Conclusion


Typed languages (including Java) you should use Either.
Dynamic languages the best would be pipeline with exceptions.



Tuesday, September 10, 2019

How to solve it? Crafting web apps using finite state machines - Part I

Code: https://gitlab.com/danbunea/how-to-solve-it-crafting-web-apps
Presentation: http://www.danbunea.ro/presentations/crafting%20web%20apps.html

PROBLEM


How should you start crafting a web app?


The requirements come, you have some UI mockups and now you need to start developing a web app. What is the first step you need to do?

Let's go back a bit. If this was not a specific web app, but rather generic:

  • What would our strategy be like?
  • What would be our goals?
  • Which steps would we have to do?
  • Which patterns could help us?
  • What tactics could we use?

In the following lines I will try to address exactly this questions. First we'll discuss the theory and then we'll develop a web app.

we'll start with a little bit of theory, then we'll put it in practice:



THEORY 

Strategy


We'll have to think a bit strategically and a little bit tactically. For the strategy, what would be our goals:

Goals

  • Break the complexity down (state machines, separation of concerns)
  • Build robustness into the system (atomicity with rollback)
Ok, but how can this be done?

Well, we'll do it in 4 Steps
  1. Finite state machine diagram (state/transitions starting from the UI)
  2. States (domain modelling for the states)
  3. Transitions (TDD the controller)
  4. Presentation/UI for the states (TDD the view)

Steps don't need to be sequential, in fact it is recommended to do the last two in parallel.

Tactics


We will be using a few Patterns:
  • finite state machines
  • separation of concerns using MVC
  • atomicity = all or nothing
and a few Techniques:
  • TDD - for the non UI
  • TDD - for the UI
  • Design by contract

PRACTICE


TodoMVC web application, because everyone loves to prove their framework with it, but we'll focus on what we said before.




Step 1 Draw the diagram of the UI state machine


So let's start with Step 1 Finite state machine

Our purpose is to create a state diagram with all the states and transitions we think we'll need. This is not an easy step, but it can clarify the entire development process further on. Let's see what can be done

Actions:
  • list
  • filter
  • add
  • check/unckeck
  • edit
  • delete
From the screens it also look like we'll have the following States:
  • list (filtered or not)
  • add
  • edit
and now let's put it in a diagram:




or:



Now using theis diagram we'll move to step 2 and we'll do the code according to it.


Step 2: Domain modelling



We will separate the different concerns using the MVC pattern. All the logic will be in the controller, the data in the model and we'll add the presentation later or someone else could do it in parallel.

Our data will have to be able to represent all the states in the diagram above: list/filter, add and edit

;What is a todo?

{:text "todo", :done? true}


;How do we know the status?
;When do we filter?


{:context {:status "list"
          :filter true}
:todos [{:text "todo 1" :done? true}
        {:text "todo 2" :done? false}]}


;What about edit?
;How do we know which do we edit?
;How do we order the todos?

{:context {:status "edit"
          :filter true
          :selected-id 1},
:todos {
        :1 {:id 1, :text "todo 1", :done? true, :time 1}
        :2 {:id 2, :text "todo 2", :done? false, :time 2}
        }}

;Let's carve it in stone:

(require '[clojure.spec.alpha :as s])

;SPEC
(s/def ::status #{"list" "edit" "add"})
(s/def ::filter #(or (false? %) (true? %)))
(s/def ::filter-with-nil #(or (nil? %) (false? %) (true? %)))
(s/def ::selected-id nat-int?)
(s/def ::context (s/keys :req-un [::status] :opt-un [::filter-id ::selected-id]))


(s/def ::id nat-int?)
(s/def ::text string?)
(s/def ::done? boolean?)
(s/def ::time number?)
(s/def ::todo (s/keys :req-un [::id ::text ::done? ::time]))
(s/def ::todos (s/map-of keyword? ::todo))

(defn edit-mode-has-a-valid-selected-id? [state]
  (if (= "edit" (get-in state [:context :status]))
    (some (into #{} (map :id (vals (:todos state)))) [(get-in state [:context :selected-id])])
    true
    ))

(s/def ::model (s/and
                 (s/keys :req-un [::context] :opt-un [::todos])
                 edit-mode-has-a-valid-selected-id?
                 ))

(comment 
(s/valid? ::model {:context {:status "edit"
          :filter true
          :selected-id 1}
:todos {
        :1 {:id 1, :text "todo 1", :done? true, :time 1}
        :2 {:id 2, :text "todo 2", :done? false, :time 2}
        }}))

Step 3: Solve the data problem


Now we know how we could represent our states as data in the model, let's test drive the different transitions which will al be functions in the controller. Normally I start with a plan, where I will know what I want to test.

The plan of the tests

 [ ] controller-should
 [ ] initialize-in-list-mode
 [ ] check&uncheck
 [ ] delete
 [ ] toggle-filters
 [ ] set a filter
 [ ] remove all filters
 [ ] set-add-mode
 [ ] save-a-new-todo
 [ ] set-list-mode
 [ ] set-one-for-edit
 [ ] save-a-changed-todo

Then we'll write the first one:

(deftest initialize-in-list-mode
         (is (= {:context {:status "list"} :todos   {}}
                (init!))))

which will obviously

Test fails! Good... now we write the code to make it pass:

;THE MODEL
(def model (atom {:context {:status "list"}}))



;THE CONTROLLER
(defn commit! [value atom-to-change]
  (reset! atom-to-change value))

(defn init! []
  (-> @model
    (assoc :todos {})
    (assoc-in [:context :status] "list")
    (commit! model)))



;THE TEST
(deftest initialize-in-list-mode
         (is (= {:context {:status "list"} :todos {}}
                (init!))))


Or:


Any refactorings? No, let's move to the second test and so on until we solve the entire data problem, one test at a time, making sure it fails, then making it pass, then refactoring the code. We end up with this:

The code is merged here:
https://gitlab.com/danbunea/how-to-solve-it-crafting-web-apps/merge_requests/1/diffs

The final code:
https://gitlab.com/danbunea/how-to-solve-it-crafting-web-apps/blob/master/test/todomvc/controller_should.cljs
https://gitlab.com/danbunea/how-to-solve-it-crafting-web-apps/blob/master/src/todomvc/controller.cljs

You'll probably be able to see that the names of the functions in our controller closely follow our transitions from the diagram.



Solve the data problem (TDD the model/controller) Done!

Now let's have another look again to see the correspondence between the transitions and the controller functions:





while the data in the model corresponds to:



Step 4: Build the UI


We will need now to make sure that our states can be represented on the screen.

For instance when we're in list mode, filtered:



How about edit state:



Now that we know how the data corresponds to the UI,
we can move on to doing the actual UI. We'll use react/reagent to break the UI into components:


Then we'll test drive the entire UI starting by planning the tests:

 [ ] views_should
 [ ] ender-the-main-screen
 [ ] render all sections
 [ ] not render main section when no todos
 [ ] render-input-text-component
 [ ] use-keys-on-input-text-component
 [ ] write something and hit enter
 [ ] hitting enter with no text
 [ ] hitting esc
 [ ] render todo input component
 [ ] add-a-new-todo-when-clicking-enter-or-go-to-list-on-escape
 [ ] render-todos-list-component
 [ ] no filter
 [ ] filter active
 [ ] filter completed
 [ ] render-todo-item-component
 [ ] normal mode
 [ ] completed mode
 [ ] editing mode
 [ ] toggle a todo item component
 [ ] render-todos-count-component
 [ ] no item left
 [ ] 1 item left
 [ ] 2 items left
 [ ] render-todos-filters-component
 [ ] render no filter
 [ ] render filter active
 [ ] render filter completed
 [ ] invoke-controller-filter-when-clicking-filters-in-todos-filters-component

It is worth mentioning that TDD-ing the UI can be split into two types of tests:
- render tests and
- interaction tests

For the render tests you send some data to your components, render it then check how it's rendered.
We start with a render test (we'll use enzyme):

(deftest render-the-main-screen
  (testing "render all sections"
    (let [component [views/screen-component data/one-todo-list-mode-no-filter]
          mounted (->> (r/as-element component)
                       (.mount js/enzyme))]
      (is (= 1 (-> mounted (.find ".header") .-length)))
      (is (= 1 (-> mounted (.find ".main") .-length)))
      (is (= 1 (-> mounted (.find ".footer") .-length)))
      (is (= 1 (-> mounted (.find "li.todo") .-length)))
      (is (= "0 items left" (-> mounted (.find "strong") .getDOMNode .-innerText)))
      )))

the test fails, so then we start writing our first component, making it pass, refactoring etc.

For the interaction tests, you render some data, then click a button and make sure the mocked function that should be invoked is actually invoked:

(deftest invoke-controller-filter-when-clicking-filters-in-todos-filters-component
  (let [invocations (atom [])
        component [views/todos-filters-component nil]
        mounted (->> (r/as-element component)
                     (.mount js/enzyme))]
    (with-redefs [controller/filter! #(swap! invocations conj [%])]
                 (testing "unfilter"
                   (reset! invocations [])
                   (-> mounted
                       (.find "#all")
                       (.simulate "click"))
                   (is (= [[nil]] @invocations)))
                ))

The code is merged here:
https://gitlab.com/danbunea/how-to-solve-it-crafting-web-apps/merge_requests/2/diffs

The final code:
https://gitlab.com/danbunea/how-to-solve-it-crafting-web-apps/blob/master/test/todomvc/views_should.cljs
https://gitlab.com/danbunea/how-to-solve-it-crafting-web-apps/blob/master/src/todomvc/views.cljs




Goals revised


Complexity problem -solved by making a state machine and using separation of concerns.
Robustness problem - TDD is a clear step forward but we can go further:

  • add atomicity into the transitions/operations (all or nothing)
  • design by contract, always making sure the states are valid states
But we'll see that in part II


Monday, July 22, 2019

Thinking in Reagent as a reaction to Thinking in React. A 1:1 transformation from Javascript/React to ClojureScript/Reagent

This article: https://reactjs.org/docs/thinking-in-react.html introduces people to working with React in Javascript. I thought it would be nice to have the same simple application made in Clojurescript with Reagent (which also uses React underneath)

For example

class ProductCategoryRow extends React.Component {
  render() {
    const category = this.props.category;
    return (
     
       
          {category}
       
     
    );
  }

}


In Reagent:

(defn product-category-row-component [category]
  [:tr
   [:th
    {:col-span 2}
    category]])


 I made an interactive presentation and I put it here:

http://www.danbunea.ro/presentations/thinking%20in%20reagent.html




Friday, September 21, 2018

Outside-In TDD for the functional python server with Flask and SqlAlchemy




In the following video, you will see a graph and how in time we expand the code using the Outside-in TDD 

Requirements


We have an web API where we should be able to write and read articles and their comments.

Acceptance Test

We save an article:

POST /api/1/save/Article
{
"title":"New article",
"content":"The content"
}
and we receive the article + id
{
id: 13, - can be any value
title:"New article",
content:"The content"
}
We then add 2 new comments:
POST /api/1/save/Article
{
id: 13, - can be any value
comments:[
    {"comment":"This was awesome!"},
    {"comment":"I loved it as well!"},
]
}
response:
{
id: 13, - can be any value
title:"New article",
content:"The content"
}
and now reading the articles and the comments:
POST api/1/query/Article
{
    find: [id,title,content,{comments:[id,comment]}],
    where:{id:13}
}
should return
{"13":{

    id: 13, - can be any value
    comments:{
    "113":{"comment":"This was awesome!"},
    "114":{"comment":"I loved it as well!"},
    }
   }
   }

Outside-In TDD approach


Unlike classicist TDD, in Outside-In, you start the code from the outside, designing it along the way. You start with an acceptance test, then with the unit/integration tests and code needed until you make the acceptance test work:




So let's begin:

Step 1: Acceptance test






When we run it:




2. Step 2 moving in, 


From the outside, writing the first test for the service that will handle the url. A little bit of design, while writing the test: we consider, that we will use a function that will get the object from the database if we have an id in the json or will give us a brand new Article object if we have no id in the json.













Now if we properly do the code, the test will pass:








Step 3: moving furher in, database_services and get_database_object


However, we did not implement the get_database_object, which will work with the database. So for that we'll write the tests first:


or:



and






Now let's write the code to make it pass:







We write in fact 2 tests to cover both scenarios:  when editing or when adding a new article.


Step 4: now we move back and extend the api


We design it to read the data from json and transfer it to the database object. For that we'll use a function transfer_from_json





Once it passes, we move further:

Step 5: writing the transfer service













Now we make the test pass:





Step 6: back to extending the api

We now need to save the object with the data from the json into the db



For this, we extend the test:



and make it pass, then we move back to implementing using TDD the save_database_object

Step 7 extending the database_service to also save



Once we write the test, the code and make it pass:



we move back to the api

Step 7 extend the api





Step 8: TDD the extension:



Until we make it pass:




After a few more intermediary steps:

...

We will TDD all the necessary code to make the acceptance test pass: