Graphical Linear Algebra Post 5 on 2019-02-18

This is the 5th post in a series of daily posts on my Timeboxed Challenge to learn Graphical Linear Algebra (GLA). The source material would be the 30 episodes from the Graphical Linear Algebra blog. If you prefer to read my series from the beginning, start here with the first post. Today, I’m re-examining my end goal for my Timeboxed Challenge itself.


I spent the last 3 days rethinking my end goal and strategy for my Timeboxed Challenge. I also spent a considerable amount of time skimming through all the episodes in the GLA blog. I find myself drawn to some key points about the philosophy of the blog itself. But also find myself losing interest as the math piled on in the later episodes.

Given that my pace has dropped considerably, there’s no way I can cover all 30 episodes by end of the month, so I have to rethink my purpose and what a successful challenge looks like.

My Motivation to Learn Graphical Linear Algebra Relation to My Work

In the past, I didn’t have a clear direction what I wanted to work on in my career. I simply go where the trends take me. Since 2018, I have resolved to have more structure in my life and work. With that aim in mind, I have become more purposeful even in my usual nomadic style of reading. I was searching for a more meaningful direction to take my software development career into.

Recently, my nomadic reading led me to Workflow And Decision Automation, henceforth known as WADA. My interest in graphical linear algebra was because it reminded me of the Lego bricks I grew up playing with. I foresee that WADA is a possible area to benefit from having Lego-brick-like structure. Or in more technical terms, compostionality.

Now that I have finished skimming through all 30 episodes of the blog, I have come to realize even if I did grok the math behind this, I may not be able to apply it immediately in my work on WADA. Which brings me to the goal I want to further design for this Timeboxed Challenge.

Goal (Re-)Design for the Timeboxed Challenge

The past week, I have been working towards the challenge in a typical forwards orientation. I start from where I am, which is knowing nothing about GLA, and then try to figure out a way towards a goal. Now, I want to work in a backwards orientation. I discover in episode 24, that Pawel was trying to lead his readers to understand all the interacting Hopf monoids he and his co-authors come up with. Below is the full list.

The full list of monoids Pawel was trying to teach

With this in mind, it becomes easier to design a more specific end point for the Timeboxed Challenge. Instead of the original goal of writing a 1 page summary of the 30 episodes of the GLA blog, I would redesign that goal in a different way. I aim to write a cheatsheet. This cheatsheet should allow the least knowledgeable beginner to bootstrap basic understanding of these monoids in the shortest amount of time possible.

This might mean the cheatsheet is less interesting than Pawel’s own writing. But, I imagine the cheatsheet would serve as an easy reference point for those who have finished reading Pawel’s episodes. Which would be the purpose.

With that in mind, I expect to change my strategy for this challenge this week.

This is post #11 in my quest for publishing weekly.

As usual, all diagrams that are shown here were taken directly from Pawel’s blog at graphicallinearalgebra.net. All credit goes to him regarding the explanation of the idea. I merely gave my take on his explanation.

Graphical Linear Algebra Post 4 on 2019-02-14

This is the 4th post in a series of daily posts on my Timeboxed Challenge to learn Graphical Linear Algebra (GLA). The source material would be the 30 episodes from the Graphical Linear Algebra blog. If you prefer to read my series from the beginning, start here with the first post. For today’s post, I will cover the concepts of Associativity, Zero, and Unit. 


Still the same Episode 5 – Spoilers, Adding (Part 2) and Zero

From the last post we ended talking about how composition acts on generators. The last example I gave was this one.

Fig 1

Pawel covered more examples in Episode 5 which I will replicate here. I wrote this series primarily more for my own learning. To consolidate my understanding better of this, I will replicate more of the examples Pawel showed in the same episode.

More Examples of Composition

Fig 2.1 (3,2) ; (2,1) = (3,1)

and

Fig 2.2 Also (3,2) ; (2,1) = (3,1)

This brings us to the next thing Pawel wanted his readers to learn.

Associativity

Associativity is just a technical term to describe where you put the brackets doesn’t matter for that operation. The classic example is addition. From my own memory of my secondary school mathematics textbook,

(x + y) + z = x + (y + z)

In fact, Pawel used the same concept involving addition to explain the associativity of composition with more diagrams.

The key diagrams are:

Fig 3.1 Proving the associativity based on the previous 2 diagrams

Using the (x + y) + z = x + (y + z) example to hammer in the associativity concept.

Fig 3.2 This is (x + y) + z
Fig 3.3 This is x + (y + z)

Just like how for Commutativity, Pawel showed a (I called it a blackboxed representation) of Associativity.

Fig 4.1 A reminder of the “blackboxed” representation of commutativity
Fig 4.2 Similarly, this is for associativity

Zero

Pawel goes into a long spiel about how zero is one of those rare, great scientific breakthroughs. Once again, I enjoyed those long spiels very much. But because, this series of posts aims to focus only on the absolute essentials. I have to skip all of that.

The key thing about zero, Pawel told us, is that it’s an additive identity.

Zero is the Additive Identity

Fig 5: A graphical representation of zero

And how it works can be best understood with an example.

Fig 6.1 : How zero as part of a generator to equal (1,1)
Fig 6.2: Same generator but with algebra in it

Unit – the Third Equation About Addition

Just like how we have the diagrams for Commutativity, and Associativity, the diagram 6.1 shows Unit. Now, I feel that this should be made clearer. And it doesn’t help that I have never come across this term in my old mathematics textbook. So, this is one of those “just accept it as it is for now” concepts.

In fact, Pawel says that Commutativity, Associativity, and Unit are the three equations we need to learn about addition. He also adds another point after that. Any structure (what’s an example of structure?) that satisfies the three properties of Commutativity, Associativity, and Unit are called commutative monoids. There is a link to the wikipedia definition to commutative monoids. But, trust me. It doesn’t really explain anything if you don’t math well. And I don’t math well.

Conclusion and Looking Ahead

Let’s sum up what I have learned in today’s post.

  • Associativity via examples of composition
  • The diagram for Associativity
  • Zero is the additive identity
  • The diagram for Unit (though I’m still not sure what Unit means)
  • The term commutative monoid
  • And that it’s the name for structures that satisfies all three properties of Commutativity, Associativity, and Unit

Pawel ends Episode 5 talking about using only diagrams to prove the following.

Fig 7: The upside down Unit.

But, when I look ahead at Episode 6, I must say I find his examples involving a cooking recipe far more interesting than the proof of Fig 7. I guess, that has more to do with me being a developer. In any case, I will study that in greater detail for tomorrow. See you tomorrow.

This is post #10 in my quest for publishing weekly.

As usual, all diagrams that are shown here were taken directly from Pawel’s blog at graphicallinearalgebra.net. All credit goes to him regarding the explanation of the idea. I merely gave my take on his explanation.

Graphical Linear Algebra Post 3 on 2019-02-13

This is the 3rd post in a series of daily posts on my Timeboxed Challenge to learn Graphical Linear Algebra (GLA). The source material would be the 30 episodes from the Graphical Linear Algebra blog. If you prefer to read my series from the beginning, start here with the first post. For today’s post, I will cover generators, special generator names, and how direct sum and composition act on generators.


Episode 5 – Spoilers, Adding (Part 2) and Zero

I have limited time today to read and study Episode 5 completely. Most probably I have to split Episode 5 into at least two separate days of posts. Today’s post will specifically introduce two technical terms — generator and identity. Adding to it, we will see how direct sum and composition affect generators.

Generators — Another Name for Basic Diagrams

In my last post (the one with dozens of Lego pictures), I learned from Pawel Lego bricks are useful for learning about GLA. Well, they were precursors to the basic diagrams in GLA. One such basic diagram or generator, as they are called, looks familiar.

Fig 1: A basic diagram or generator

We actually have seen this in Episode 3 before. I have reproduced this as a notation in Post 1 – On Matrix Notation and Dangling Wires to explain our normal arithmetic addition sum.

Fig 2: Remember this from Post 1?

So this is a generator. Remember it. Though I must say, Pawel did not explicitly state what diagram is NOT a generator. For now, let us mothball that question. Perhaps it will be answered in future episodes.

Special Generator Names

Now, we have specific names for specific kinds of generators. Not too sure how useful to learn the names, but we’ll just go with it for now.

Identity – Just Your Typical (1, 1) Generator

When you have a Magic Lego brick with 1 hole and 1 stud, this is an identity. Well, I mean, a generator that’s (1, 1).

Fig 3: Identity

Twist – The (2, 2) Generator that Literally has a Twist

And a (2, 2) generator called a Twist. Which is aptly named because it has a twist in its middle.

Fig 4: Twist

How Direct Sum Affects Generators

If you can recall how Direct Sum affects our Magic Lego bricks, you should have no issue with using Direct Sum on generators. It’s exactly what you expect.

Fig 5.1 One way of using Direct Sum
Fig 5.2 Here’s another. Proving that direct sum is NOT commutative

How Composition Affects Generators

Again, just recall how composition works on Magic Lego bricks and you’ll be fine.

Fig 6

A gentle reminder about composition. The left studs must match the right holes for composition to take place.

Another reminder. As long as two diagrams share the same number of holes and studs, they are considered as equal.

Fig 7 These two diagrams are equal based on holes and studs

Conclusion and Looking ahead

A quick summary:

  1. A basic diagram is called a generator
  2. Generators are what the Magic Lego bricks were referring to. For the past few episodes.
  3. A (1, 1) generator is called an identity.
  4. A (2, 2) generator with a twist in the middle is called a twist.
  5. If you recall direct sum on Magic Lego, it works the same way on generators.
  6. If you recall composition on Magic Lego, it works the same way on generators.
  7. Gentle reminder 1: composition has a condition where the left studs match the right holes.
  8. Gentle reminder 2: two diagrams are equal when they are said to have same number of holes and studs.

Looking ahead, the last part of Episode 5 is covering more composition and a new property called associativity. Finally, it ends with an explanation about zero. I’m looking forward to covering that in tomorrow’s post.

This is post #9 in my quest for publishing weekly.

As usual, all diagrams that are shown here were taken directly from Pawel’s blog at graphicallinearalgebra.net. All credit goes to him regarding the explanation of the idea. I merely gave my take on his explanation.

Graphical Linear Algebra Post 2 on 2019-02-12

This is the second day of my Timeboxed Challenge. On the first day, I summarized the first three episodes in the GraphicalLinearAlgebra blog. I doubt I will cover as many episodes in today’s post. In episode four on Magic Lego, Pawel spent the first 500 words addressing complaints about his writing style. In essence, people want him to be more concise. Shorter. Personally, I enjoyed his digression from time to time. It helps break up the monotony of dense material. Pawel also mentioned about the London Review of Books which he subscribed to. His admiration is clear in his own words here.

The authors, who are usually domain experts, make their subjects extremely clear to a lay audience, but without dumbing them down. They take all the space they need and they respect their readers.

That’s also a sentiment I admire greatly. If I get anywhere near that level of writing, I’ll be hugely pleased. Hence, my publish weekly challenge 2019 as part of the Year of Consistency theme to get better at writing.

They take all the space they need and they respect their readers.That’s also a sentiment I admire greatly. If I get anywhere near that level of writing, I’ll be hugely pleased. Hence, my publish weekly challenge 2019 as part of the Year of Consistency theme to get better at writing.

Episode 4 – Dumbing Down and Magic Lego

This episode is centered around Magic Lego bricks. I assume these Magic Lego bricks are a bridging tool into the Graphical Linear Algebra (GLA) concepts. When I completed this episode, Pawel did not reveal how these Magic Lego bricks are analogous to GLA concepts. I expect Pawel to continue using Magic Lego bricks for some time.

Because this is a summary, I will be more concise than Pawel. Read his post if you prefer the whole treatment. In essence, via Magic Lego bricks, he wants to teach you six concepts.

Concept 1: An Operator Called Direct Sum

Pawel explains direct sum (represented by this symbol ‘⊕’) using this example.

Fig 1
Fig 2

The way I understand this is simply that direct sum ⊕ is about joining bricks in the vertical dimension. And that the order of the arguments matters. In fact, Pawel emphasized the importance of the arguments order as well. He pointed out that direct sum ⊕ is NOT commutative. Unlike the + operation we learn in previous episodes.

In case you are unsure, arguments refer to the values for any operation. Recall 3 + 4 = 7 example in my last post? So arguments for + operation are 3 and 4. Back to our article.

Concept 2: A Second Operator Called Composition

Direct sum is an operator. Pawel then covers a second operator called composition (represented by ‘;’).

Fig 3
Fig 4

In Fig 3 and 4, I see that composition is an operator that acts on the bricks in horizontal dimension. Bear in mind, the use of the phrase “vertical and horizontal” are entirely my own paraphrasing.

Once again, this operator composition is also NOT commutative.

Concept 3: A Magic Lego Brick can be Represented as Holes and Studs

You can represent a Magic Lego brick as a set of two numbers (holes, studs). To remember which number comes first, I recommend remembering it as alphabetical order. Since holes is alphabetically before studs, you get (holes, studs). Fig 5 below shows us how different bricks are represented in this format.

Fig 5

Concept 4: When Compose is Allowed and Forbidden

After explaining the 3 concepts, Pawel introduced the notation for direct sum and composition. I’m not going to follow that order. I prefer the visual examples of the Lego bricks, so I’m going to continue that for a while longer. The next thing to understand is that certain situations, we cannot perform compositions.

Fig 6

In Fig 6, we have a Lego brick that has two studs on the left. On the right, we have four holes. This is not allowed in composition. In deliberately broken english, I will summarize this way. Composition is allowed when left studs equals right holes.

This point is so important. I want to repeat it.

Composition is allowed when left studs equals right holes.

This means we have some weird situations when composition is allowed.

Allowed Compositions

Fig 7

Fig 7 is allowed. We have two studs on the left and two holes on the right. So this composition is allowed. What the outcome looks like has two possibilities.

Fig 7.1
Fig 7.2

Both outcomes are the same. This is where the Magic in Magic Lego is referring to.

Fig 8: Both outcomes are the same thing

The Magic-ness is not limited to composition. Which brings us to the next concept.

Concept 5: Direct Sum also has Magic Outcomes

Fig 9

This is also allowed in direct sum. The Magic works in the sense that we can’t tell the difference between plates and bricks.

Concept 6: Representing Direct Sum in Notation

Now, we’re ready to talk notation! Direct Sum is represented in this notation.

Fig 10: The rule for Direct Sum

Let’s go through this part by part. Notice how this has a top part (above the line) and bottom part (below the line). The top part represents assumptions. The bottom part represents conclusions.

Assumptions

Let’s go through the assumptions first. Reading from left to right, we understand that we have two bricks, X and Y. X has k holes and l studs. Similarly, Y has m holes and n studs. That’s basically it.

Conclusions

So when you read the whole formula, it becomes assumptions such that conclusions. We have :

  • X with k holes and l studs and Y with m holes and n studs
  • such that
  • when X direct sum Y, we get a new brick with (k+m) holes and (l+n) studs

Concept 7: Representing Composition in Notation

If you skip the previous concept, please go back. Else, this is pretty much the same idea. Start with the assumptions and then work your way through the conclusions.

Fig 11: The rule for Composition

One thing I like to bring your attention to is the fact that the stud in X is the same as the holes in Y. Both are represented by the letter l. This is exactly the concept I covered in Concept 4 earlier. For completeness sake, I will explain the notation from top to bottom again. We have:

  • X with k holes and l studs and Y with l holes and m studs
  • such that
  • when X composition Y, we get a new brick with k holes and m studs

Conclusion

Rounding this off, we learn that:

  1. There’s an operator called Direct Sum and it’s not commutative.
  2. A second operator called Composition and it’s also not commutative.
  3. A Lego Brick is represented by (Holes, Studs).
  4. Composition is allowed only between X and Y when the studs in X is the same number as the holes in Y.
  5. There’s no difference between plates and bricks in Magic Lego Bricks combining through Direct Sum or Composition.
  6. The notation that explains Direct Sum between X and Y bricks producing a new brick whose holes and studs are the sum of the holes and studs of X and Y.
  7. The notation that explains Composition between X and Y bricks that need X studs to be the same number as Y holes. This will produce a new brick that has the holes of X and the studs of Y.

See you tomorrow!

This is post #8 in my quest for publishing weekly.

As usual, all diagrams that are shown here were taken directly from Pawel’s blog at graphicallinearalgebra.net. All credit goes to him regarding the explanation of the idea. I merely gave my take on his explanation.

Graphical Linear Algebra Post 1 on 2019-02-11

Due to unexpected circumstances, I couldn’t spend more time studying the GraphicalLinearAlgebra blog. Thankfully, the first two episodes were your typical introductory material. So they touched on big picture stuff about linear algebra and category theory. I was able to peruse them quickly. I was also able to cover what a matrix is, what a graphical notation is, and the concept of dangling wires.

Episode 1 – Makelele and Linear Algebra

In the first episode, it compares linear algebra with a famous footballer ( Claude Makélélé) in a more famous team (Real Madrid). The analogy was to point out that linear algebra is the highly critical yet unsung hero in the field of science and mathematics. Much like Makélélé.

Then, it introduces matrix to end the episode. A matrix is, basically, a double array of numbers. We describe the matrix as a row x column. So if it’s 2×3 matrix, it’s a matrix with 2 rows and 3 columns producing 6 numbers.

A 2×3 matrix

Episode 2 – Methodology, Handwaving, and Diagrams

In the second episode, Pawel Sobocinski goes into more detail about his plans for the series. There are only a few key points I like about this episode that I want to highlight.

  1. He promised not to “handwave“. In fact, he asked for specific feedback when he does indulge in handwaving. Which is an admirable stance.
  2. He forewarned about having lots of diagrams. Hence, the “graphical” in Graphical Linear Algebra. Pawel promised how in future episodes, a diagram can even tell a thousand formulas.
  3. Pawel mentioned about category theory. He even added that “category theorists are comfortable with diagrams”. And going forward, he will “use category theory to do graphical linear algebra.” Not sure what that means exactly, as I have only the vaguest notion of what category theory does.

So far so good, all simple stuff. Both episodes can be glanced through without any significant loss of understanding. Now, we come to the first real challenge of learning Graphical Linear Algebra in Episode 3.

Episode 3 – Adding (Part 1) and Mr Fibonacci

In this episode, Pawel used the example of a simple addition formula to illustrate the use of graphical notation. As I’m writing this, I also found myself going back and forth between the blog and this draft. So if this sounds more like sports commentary than a proper summary, do forgive me.

Back to the summary. Pawel started with showing simple mathematics formula can be represented graphically like this.

3 + 4 = 7

Notation for 3 + 4 = 7

And then he digresses into a story on Fibonacci and the Hindu-Arabic numeric system. Which I rather enjoy to be honest.
Because I’m assuming we’re going at the level of the freshman lecture, let me be direct here. In this episode, Pawel is trying to teach us the notation for commutativity via addition.

Commutativity is this fancy term to say that

∀ x,y.  x + y = y + x   

Another way to say this out loud is for all values of x and y such that, x + y = y + x.

And the notation for the above formula can be represented as:

This means the same as ∀ x,y.  x + y = y + x   

Then Pawel did highlight one key point for graphical linear algebra.He mentioned when we have an equation between two diagrams, the number of dangling wires on the left and right of both diagrams must be the same.

If we reference the last notation, we can see how that is true. On the left diagram, we have 2 dangling wires on the left and 1 on the right. For the right diagram, we have the exact same number of dangling wires on the left and right of that diagram.

Note the left diagram and the right diagram have the same number of dangling wires

Generalized form of addition

Finally, Pawel appears to be teaching us a general form of the diagram in the Comm equation. He says that both the left and right diagrams with their dangling wires can be presented as:

This has 2 left dangling wires and 1 right dangling wire. Same as the diagrams in the above equation.

Looking ahead to Episode 4

That’s it for now. Tomorrow, I will continue with another episode. I have previewed the next episode ahead of tomorrow’s summary. It appears that I won’t be able to compress multiple episodes into 1 blogpost summary like I did here anymore. In fact, I may need posts stretching over several days to summarize my thoughts for one episode going forward.

I’ll cross that bridge when i get there. In the meantime, thank you for reading.

This is post #7 in my quest for publishing weekly.

All diagrams that are shown here were taken directly from Pawel’s blog at graphicallinearalgebra.net. All credit goes to him regarding the explanation of the idea. I merely gave my take on his explanation.

Announcing the Timeboxed Challenge

I came across this Month to Master series by Max Deutsch during my recent Lunar New Year holidays. And I found it intriguing. You can dig deeper, but the high level idea is that Max will pick a challenge to master a skill within a month. He’ll also blog about his progress on a daily basis.
He has already completed 1 full year of 12 challenges in 12 months. Some of these are insanely tough. Like this one on How to build a self-driving car in one month. In any case, the idea of a timeboxed challenge seems to fit right in with my systems-obsessed brain.

One Stone, Many Birds

I haven’t really thought this through. It may turn out to be a bad idea. Currently, I am busy with many projects at the same time. I run a full-time software business in workflow and decision automation. I’m aiming to write a blogpost every week as part of my Year of Consistency theme. And there’s a weekly newsletter I’m writing as well. Not to forget, my addiction to reading and learning new stuff. Which includes topics such as design and Javascript that I intend to help me in my software business.
Given that I have not blogged two weeks, I want to restart my blogging momentum. And I’m doing that by (wilfully) adding more to my already packed load.
I like to take all my problems and lump them together. 😀

The First Challenge

So this post will signal the start of a series of timeboxed challenge. The first challenge is to produce a Graphical Linear Algebra for Humans post.
Some background information. I came across this field called Applied Category Theory a year ago. I tried reading free resources about it to understand more. But I have made no progress whatsoever. It piqued my interest because it reminded me of the Lego toys I played in my childhood. And in a sense, I’ve been trying to replicate the joy I had with Lego in my work building systems for enterprises.

Applied Category Theory

Recently, and by that I mean today, I came across Applied Category Theory (henceforth ACT) again. I joined a telegram chat group started by StateBox. Statebox appears to be a group (company?) dedicated to building composable programming language. In other words, they want to build a programming language for developers to use it like Lego bricks.
One thing led to the next. And I came across something that looked a lot simpler than the free ACT resources I discovered a year ago. A website trying to explain Graphical Linear Algebra in 30 “episodes”.

Feynman Test of Understanding

Hence, my challenge is to create a summary of those 30 episodes. A summary so simple that a freshman can understand it. Basically, it’s a test of true understanding. The idea was popularized by Richard Feynman, a Nobel prize-winning scientist. He firmly believes that if you cannot explain in simple terms, you don’t really undertstand it. I call this the Feynman test.
I pick the level of freshman because that’s the level Feynman sets for himself.

The Logistics

Today is already the 10th of February. I’m giving myself only till end of February anyway to complete the challenge. Another logistical detail to add is that I will write a post every day on my progress anyway. This will force me to make sure I post something worthwhile every day. Thus, ensuring that I will, at the least, achieve my weekly post habit I have now started to struggle with.

Let the Challenge Begin!


In summary,

  1. I’m setting a Timeboxed Challenge every month starting February 2019.
  2. Beginning with producing a freshman-level summary of Graphical Linear Algebra post for February
  3. While posting every day about my progress

This is post #6 in my quest for publishing weekly.

How to check if item in list/array (Python and Javascript)

I’m going to try something new this week. I’ve always had the intention of writing down a series of common syntax for Python and Javascript. Because I keep looking them up anyway. So I thought, this would be a good opportunity to try writing them out as two separate Series. One for the Weekly Post and one for How to X in Python and Javascript. In this post, I will touch on how to find an item in a list.

Situation: You have a list (Python’s preferred term) or array (Javascript’s preferred term) of items. For a simple example, let’s just use a list of numbers [1, 2, 3]. You want to find out if the number 3 is in the list. Hence, you need a boolean result.

Python

3 in [1, 2, 3] 

Javascript

arr = [1, 2, 3]
arr.includes(3)

This is post #5 in my quest for publishing weekly.

Photo by Sophia Baboolal on Unsplash

How to Restore Database Dumps for Postgres in Docker Container

Before 2018, I largely use the following technologies. MySQL for database and VirtualBox and Vagrant for running my environments. Starting in mid 2018, I moved towards using Postgres and Docker instead. This is especially the case when I paid to learn about cookiecutter-django. Adding Postgres and Docker to my toolbox has increased my effectiveness in my software business. However, this also means I need to figure out regularly used tasks involving these new technologies. Some frequent tasks includes backing up and restoring database dumps. Therefore, this article is about how to restore database dumps for Postgres running inside Docker containers. I’ll write the counterpart article to the backup process on a separate day and add the link to it when it’s ready.

Key Insights You Need to Know About Docker Containers

Before I go into the step by step about the backup and restore database dumps, you need to first acquire some fundamental insights of the Postgres and Docker technologies.

  1. Firstly, Docker containers have their own volumes. Think of them like the disk volumes in your host system.
  2. Next, realize that executing commands inside the docker container from your host system is possible. Which is to run docker exec <container_name> <your_command>
  3. When you want to run certain commands within the containers and these commands need to interact with certain files, the assumption is that these files are found in the docker container’s own volumes.

Therefore, when you execute the typical Postgrespg_restore commands, the database dumps will need to be in the docker container’s volumes. This is key.

There are several ways to achieve the transfer of files between your host system and the docker container. Similarly, there are several ways to run backup and restore. To keep things simple for beginners, I will only state one way to accomplish this. It doesn’t mean my one way is the best way. But, keeping it to only one way makes it easy for beginners to follow. Moreover, it’s easy for me to update the content here as time goes by.

How to Restore Data Dump Using pg_restore

Step 1: Find the name and id of the Docker container hosting the Postgres instance

Turn on your Docker and run the docker ps command to locate the name and id of the Docker container. Which leads to the following.

$ docker ps

CONTAINER ID   ...                  NAMES
 abc985ddffcf  ...              my_postgres_1

Step 2: Find the volumes available in the Docker container

Run the command docker inspect -f '{{ json .Mounts }}' <container_id> | python -m json.tool

Then, look at the volume paths under the key Destination.
You should get the following:

$ docker inspect -f '{{ json .Mounts }}' abc985ddffcf | python -m json.tool
[
    {
        "Type": "volume",
        "Name": "my_postgres_backup_local",
        "Source": "/var/lib/docker/volumes/my_postgres_backup_local/_data",
        "Destination": "/backups",
        "Driver": "local",
        "Mode": "rw",
        "RW": true,
        "Propagation": ""
    },
    {
        "Type": "volume",
        "Name": "my_postgres_data_local",
        "Source": "/var/lib/docker/volumes/my_postgres_data_local/_data",
        "Destination": "/var/lib/postgresql/data",
        "Driver": "local",
        "Mode": "rw",
        "RW": true,
        "Propagation": ""
    }
]

In this case, we have /backups and /var/lib/postgresql/data as the volume paths.

Step 3: Copy dump into one of the volumes

Pick a volume and copy your dump in. Run docker cp </path/to/dump/in/host> <container_name>:<path_to_volume>

In my case, I pick the volume /backups. Which gives us the following.

$ docker cp my_data.dump my_postgres_1:/backups

Step 4: Get the database owner to run pg_restore command

Execute the pg_restore command via the docker exec command. Which means the generic forms of both commands are the following.

Note: I’m assuming that the destination postgres database already exists.

For pg_restore:

pg_restore -U <database_owner> -d <database_name> <path_to_dump>

For docker exec:

docker exec <container_name> <some_command>

Sometimes, you don’t know who the database owner is. This is purely optional if you already know. In any case, you can find the owner by retrieving the list of databases and their owners. That means a psql -U postgres -lcommand. Which you have to run within the docker exec command as well. Therefore, we get the following.

docker exec my_postgres_1 psql -U postgres -l

          List of databases
        Name        |  Owner   
--------------------+----------
 some_database      | postgres 

After I have all the information I need, I’m ready to run pg_restore. Then, this becomes the following.

docker exec my_postgres_1 pg_restore -U postgres -d some_database /backups/my_data.dump

Conclusion

I started by covering some key fundamentals about Docker and Postgres. Following which, I went into code level details about the commands to restore your Postgres data dump in a Docker container for Postgres.

Now, I end with a summary of the four steps to do so.

How to restore Postgres data dump in a Docker container using pg_restore

  1. Find the name and id of the Docker container hosting the Postgres instance

    Turn on Docker and run docker ps to see the list of containers and their names and ids.

  2. Find the volumes available in the Docker container

    Run docker inspect -f '{{ json .Mounts }}' <container_id> | python -m json.tool

  3. Copy the dump from your host system to one of the volumes

    Run docker cp </path/to/dump/in/host> <container_name>:<path_to_volume>

  4. Execute pg_restore via docker exec command

    docker exec <container_name> pg_restore -U <database_owner> -d <database_name> <path_to_dump>

Problems? Errors? Leave a comment below to let me know if it works for you.


This is post #4 in my quest for publishing weekly.

Photo from Youtube

The Year of Consistency

There’s a famous quote by Woody Allen that 80% of success is just showing up. That’s a great quote to help people get started. Once people get started, the formula should be changed to 80% of success is showing up consistently. Just like there’s no one catch-all universal recipe for all people under all contexts, there’s probably no one single catch-all universal recipe for the same person under all stages of their growth. I’m now 37 and life experience tells me in general there are 3 or 4 stages of growth regardless of any fields. The first stage is simply to get good at starting and restarting. In Woody Allen parlance, get good at showing up.

80 percent of SUCCESS is showing up.

“Showing Up Is 80 Percent of Life – Quote Investigator.” Accessed January 6, 2019. https://quoteinvestigator.com/2013/06/10/showing-up/.

You will start-stop-start-stop-start-stop ad infinitum. Nobody tells you this. But it’s okay to start-stop-start-stop repeatedly at first. It’s also okay to change tactics to help you start, or rather restart more easily each time you stop. It’s confusing as a beginner and reading too many expert tips just adds to the friction of starting and re-starting. Floundering is normal. All you can do is begin again. Every stop is a pause for you to recalibrate and begin again faster, smarter, or better. Nobody builds muscles overnight. Training is merely one series of starting and stopping training sessions until you start to get the hang of it. But once you are past this Start-and-Restart Stage, you acquire a sense of familiarity with the new skill you’re learning. It’s time to go to the next stage. I call that next stage — the Consistency Stage.

2018 was my Start-Restart Year

2018 was a year where I had to start from scratch. I set out to build a new lifestyle with a new direction in my business. Looking back, I now realize 2018 is my personal Start-Restart Stage. I haven’t completed reviewing my 2018 and planning for my 2019 goals even though it’s already 1 week into the new year. Am I slow compared to everybody else posting their 2018 review posts and 2019 resolutions? Yes. But, I run my own race. I don’t run theirs and other people don’t run mine.

One thing I did get clarity in my still ongoing review and planning is that since 2018 is my Start-Restart Stage for a new me. 2019 is surely my Year of Consistency for the new habits and actions I’m adopting.

Previously, I didn’t care too much about how often I was doing the new habits I wanted to form. Or even how well I performed them. So long as I keep starting and restarting, then that’s all fine and dandy. Results started to change for me. I racked up more steady revenue and started getting fitter and healthier. Towards the end of 2018, I instinctively began to track performance metrics. From the number of hours I work, to the calories I eat. Slowly, I seem to gravitate towards being and acting more consistently.

Tracking my hours spent in the last week of 2018: I spent 45 hrs working in 2018 week 52
An example of tracking new metrics: my tracker on time spent in last week of 2018

With this in mind, the theme for 2019 would be the Year of Consistency. Meaning to say, simply showing up is the old benchmark. The new benchmark is how often I’m showing up. If this sounds vague for you, I agree. But I don’t really write for you. I write to clarify my own thinking. The good thing is I can feel both my emotional and rational sides are on board with choosing Consistency as the theme for 2019.

2019 – The Year of Consistency Targets

What does this mean? I’m making changes in a few areas all at once. And I’ll share more details when I get more clarity yet. Suffice to say that at the topmost of my mind are my business, health, and blogging goals. For example, I set myself a target to publish a new post every week. Tentatively, I am setting myself targets which I have 100% control over. These targets are subject to change as I gain clarity over them. As of the first Sunday 2019-01-06, the targets and the related areas are:

  1. Publish new post every week (blogging)
  2. Work a solid 45 hours and no more on average every week (business)
  3. Maintain less than 1300 calories and protein intake of 150gm every day (health + nurition)
  4. Cardio + mini workout or major workout in gym every day (health + fitness)
  5. Execute the Start-and-Restart method for new skills I want to acquire at least once every week (business + mental well-being + knowledge)
  6. Execute the Start-and-Restart method on the Big Promises I want to fulfil at least once every week (mental well-being)

Some of the items mentioned I deliberately obscure because I’m not quite ready to disclose the details. When I’m ready, they will be made known. My blogging method is heavily influenced by my software background. I believe in an attitude of permanent beta. I expect to be tweaking continuously over time even on posts that I wrote years before. Some of these targets might change drastically or slightly. It depends as I acquire new knowledge about myself and the greater world at large.

What about you? What’s your story for 2018 and 2019? Let me know.


This is post #3 in my quest for weekly publishing.

Photo by rawpixel on Unsplash

How to Clone Multiple GitHub Repos with Deploy Keys

You have a single user account (deploy-user) on a server instance and you want to deploy multiple GitHub repositories with the same deploy key. You successfully do that for the first repo repo-first. But when you try to do that for the second repo repo-second. GitHub stops you.

In fact, when you add the same deploy key to the second repo, GitHub shows you an error message that says, GitHub gives you an error message Error: Key already in use. In their documentation, they state that

Once a key has been attached to one repository as a deploy key, it cannot be used on another repository.

“Error: Key Already in Use – User Documentation.” Accessed January 2, 2019. https://help.github.com/articles/error-key-already-in-use/#deploy-keys.

Now you can take that key and add it to your user account in GitHub instead. But that would grant read and write access to ALL the repos for deploy-user.

That’s incredibly insecure. So what do you do if you want to have deployment for your repos in the same machine but using deploy keys?

Step 1: Create different key-pairs for different repos for same server user account

This step is pretty simple. I prefer to create the different keys like this

ssh-keygen -t rsa -b 4096 -C "repo-first@servername-deploy-user"
ssh-keygen -t rsa -b 4096 -C "repo-second@servername-deploy-user"

And then I typically copy out the public keys this way:

cat /home/deploy-user/.ssh/repo-first.pub

From this, I’ll copy and add the keys to the respective repos. Typically, I also disallow write access for these keys. This time, you should successfully add the deploy keys to both repos.

Step 2: Set up the SSH Configuration Per Repo

I’ll edit the ssh configuration this way. As the deploy-user, I run vim ~/.ssh/config. It opens up the configuration file and add the configuration like this:

Host alias-repo-first github.com
  Hostname github.com
  IdentityFile /home/deploy-user/.ssh/repo-first

Host alias-repo-second github.com
  Hostname github.com
  IdentityFile /home/deploy-user/.ssh/repo-second

Why do you need this? Because when you run the git clone command, git will automatically pick the default SSH key id_rsa to attempt the connection. Therefore, we need this configuration to get around this automatic selection of the SSH key.

Note: there’s a space between the alias and github.com for each set of configuration under the Host key.

Step 3: Verify the SSH Configuration

To test this works, exit the configuration file and type the following:

ssh -T git@alias-repo-first

You should see the following if successful:

Hi Organization/repo-first! You've successfully authenticated, but GitHub does not provide shell access.

Repeat the same for each repo’s configuration.

Step 4: Clone the Repo

This is the easiest step. Run the git clone command for each repo.

git clone git@alias-repo-first:Organization/repo-first.git

That should solve the situation of cloning multiple GitHub repos with purely deploy keys.

Conclusion

Just to summarize, these are the steps.

How to Clone Multiple GitHub Repos with Deploy Keys

  1. Create 1 pair of SSH key per repo

    For example, ssh-keygen -t rsa -b 4096 -C "repo-first@servername-deploy-user"

  2. Set up SSH config file

    Indicate which key-pair for which repo. Example,
    Host alias-repo-first github.com
    Hostname github.com
    IdentityFile /home/deploy-user/.ssh/repo-first

  3. Verify the configuration

    Test your configuration using ssh -T git@alias-repo

  4. Clone the repo

    Now the moment of truth. git clone git@alias-repo-first:Organization/repo-first.git

This is post #2 since I devote to publishing every week.

Photo by Brina Blum on Unsplash