Software Architecture Tips- Don't put logs in your production DB

Sometimes ago, I had some discussions with one of my friends regarding having the logs inside DB.
While it might be interesting for many people, it is really a bad idea.

Why people decide to have their logs on DB

First thing, first :) we have to see what is good about having your logs inside a DB. Most people claim that when you have multiple servers (like web-servers) it is hard to collect your logs and combine them in order to understand what goes wrong.
That is a true point, but there are many other ways to do that, without affecting your important resources.

What is the disadvantage?

* First, you are sacrificing the most valuable resource in your system to keep some stupid logs to use in the future! You just need to have them somewhere for future reference! That is all! No realtime access, no need to have indexing or etc.

* Secondly, you might loose your logs, when the network goes down, or the DB goes down, you will not have a clue about what had happened in your code.

* Then: You will stop logging the other important parts in your code, since you will think of the resources you are using. You will loose all Debugs and Info's because if you start logging them, your DB will struggle to return a simple select to you! So you will ignore many important logs that you will need in the future.

What is the log in the view of Software Architecture

Logs are simply some data that have to be managed separately. They are not there to meet your systems functionalities. The only reason for having them, is to help you find out what is wrong with the system and fix it ASAP.

What to do then?

First, your logging system has to be implemented in a way that you can change the behavior easily. If you want to add debug logs or remove them you have to be able to do it. The thread that is handling your logs has to be different than the ones which are handling your systems process. No critical resources has to get busy because of logging.

But it is takes a lot of time!

There are lots of different logging libraries. One of the best ones are Log4net for .net applications. It has been implemented by Apache  and it is very easy to use.
It has an xml file that you can use to say what to do with a log, and what shall be the format of the output, etc. You can also say that I want my output to be handled in several different ways, and it has a separate thread so it won't affect your system.

Where shall I put my logs then?

Of course the first place to put your logs, is on the disc. It is available all the time (if not, your server will go down so you have no logs :) ), it is not a valuable resource and it is almost free since you are on the web-server. I suggest to have a file for all of your logs and another one only for Errors and Fatals, so you can see the errors easily.

But I can't check my logs everyday specially since I have several servers

As I mentioned, it is not a problem of your system. The log manager has to handle it and you can do it easily by confining your logger. There are many solutions for collecting logs. Sentry is one of the simplest. You can tell your logger to push all errors to your sentry account and then check them in a managed third party web-application easily. There are also some other systems that check your log files and update their status based on those files.

So if you have several applications that you have to take care of, you can see all of the process in a third party application somewhere else and you've just used 1 thread of your system, your web-applications disc and a small portion of your network.

Agile, Why shall I spend lots of time on brainstorming, pair-working, stand-up meeting

Introduction

If you have every worked with Agile processes like SCRUM, you will see that there are lots of time for brainstorming/ stand-up meetings and pair working. Considering all the time that will be spent some will decide to ignore some of those rules and it may have dramatic effects on the quality of the solution/ total time that has been spent on the project.

General Idea
Back in the old days, there were people who analyze the needs, then some others had to design a system based on those analysis and at last there were some programmers who would implement the code. Generally speaking, the analyst was the only one who had to be the expert on the field of the project and based on his vision, the designers (some people with university degree in software engineering)  would design a model and test it based on the analysis and the programmers just have to implement the model.

What happened in practice
* It was very hard to implement. Software teams were not mature enough to implement everything as the idea said so in practice, they spent a lot of time on design and analysis, but at the end the implemented result was far from what the customer needed. (because of the gray zone/ incomplete data/ low-depth analysis/ lack of software knowledge, etc)

*Around 60% of the time had been used for creating documents that no one would read or understand in the end. Add this time to maintenance time since the documents also have to be the updated.

*There was a big gap between analysis and implementation, so when they finished the implementation the customer might not need it anymore. (We consider it as a failure)

What About Agile

With agile we took out the over head of analysis and design. But it is not magic! nor miracle!
It simply says, choose some programmers who know basics of software design and can solve problems (we call them developers), give them authority to decide what shall they do and let them talk to the customer.

What If we remove the brainstorming session?

In Agile, Brain storming session is a meeting that your team decide about the time that they will need on each task. On SCRUM for instance, there is a session on the start of each sprint which can take up to 1 day. So many people will say: OH! the whole team for 1 day waste their time to decide about the time that they need?!!! I will ask one of them who has enough experience to do this instead, so I will win lots of time!
Well, then you will end up with a team that has no vision about the whole tasks, and therefore the code quality will decrease and you might end up with spending lots of time on merging/refactoring/maintenance instead.
Worst than that, if your expert don't describe what he had in his mind, the one that has to implement it might implement the task with another way, and he is not to blame since he have a limited vision and with that vision, his solution works fine.

What if we remove the pair working
Pair working is the process of 2 people working together on the same machine. Well, one is working and the other is sitting behind his hand to prevent mistakes.

Yup, you might say that it will double the time that you need to spend on each task, so why to work as pairs?
Well, it will prevent people from implementing a very complicated code that no one would understand, the maintenance cost will decrease since 2 people checked it line by line. And you have 2 people to handle the changes because both have the knowledge of the project.

* My experience: don't work as pairs when the task is simple, otherwise you will get board and you've wasted your time.

What if we remove the stand-up meeting

Stand-up meeting is for your team to understand what is going on while helping each other. Some would say that when we have a problem, we will ask each other, we are good friends, etc. but in practice I saw these scenarios:
*The one who is drawn in the problem: Software developers like google very much. They will try to find the answer to all problems there. While it is a good way to solving your problems, it is a wast of time if you spend 1 week to solve a problem that your teammate already knew,

*The one with lack of experience: It is really hard to ask a question for people who don't have
enough experience, because for them it is like admitting that they don't know. Also, other people are mostly busy in the day so they might be interrupted when the others ask too much questions. Therefore, it will be late when you see their code and understand what a mess they have made up.

*The one who is not Taciturn: In every team, there are some people who enjoy their inside world, more than the outside world and that is why they choose this job. If there is no stand-up, no body will understand what are they doing, their achievements and their mess up things.

*The one with complicated solutions: Software developers like to write codes that can be applied to a general issue and normally it is good, but not when they change a simple subject to a twisted and a lot harder problem that is hard to read. Of course, code review is one of the ways to prevent this problem, but it is simpler this way. Like when you see someone wants to spend 2-3 days to implement a simple service wrapper.

*The one with rondevu point:
Stand-up meetings is a rondevu point. It help the team to manage their time (developers don't like to wake up early in the morning so you might see someone who wants to start his day at 1 a.m :) )
It helps people to maintain their focus since no one will ask you a question when you are deeply in your task

Conclusion 
Software development is not a concrete block that you cannot change. You need to understand pros and cons and the application on your team in order to get the best result. So if you want to do Agile, but you want to get rid of any of the above, make sure that you know the consequences.

Agile Software Development: Are we really equal?

When people talk about agile process like Scrum, it is pretty normal to hear that there is no job title like architect or programmer, etc. so many will think that as a "developer" or a"team member" everyone has to be equal. Well :) if you think the same, you are wrong.

Agile process' usually define for a small teams (like 4-9 team members) so it is understandable to expect people to have some common knowledge, but of course that doesn't mean that you are equal even if the whole team works in a same department. 

Here I want to share some of my personal experiences with you:

Personal Experiences

It is very important to recognize the individual in the team. Each person has some special abilities and came from different background, We all write software systems but even on a single project, we have different tasks so it is clear that each one looks at the task from different aspect.

When team needs to take a decision about an issue, it is important that all attend the meeting and share their ideas, but it is predictable that someone who has some background will have more to discuss. More importantly it doesn't mean that you have to convince everyone that the idea is great. Of course if your team members are mature enough, they won't insist much to choose the easy way when the ones with more experience already chosen another one or vise versa.

As an example, I was working in a very good team and all members were senior developers and we used scrum. When there was a discussion about structure or architecture, it was people with architecture background who discussed the ideas mostly, but of course everyone shared their ideas and tries their best to improve the solution. After some discussions, sometimes one or 2 didn't convince totally but since we know each other, we just trust on the solution on experts.

The role of a "dean"

On a daily basis, a normal team can handle most situations. But there are some times that you need to ask someone to help you choose. I want to emphasis on that there is no boss or things like that in the team and when I refer to dean I don't mean someone that is superior to others in any aspect, I specially believe that a software team has to be flat and everyone has to be responsible for what he do. So what does it mean to have a dean?!
A dean is someone whom you trust, with his judgment. It doesn't even mean that the one has more knowledge than you.

For instance, we had a small project and we had to implement a service with lots of needs. The 3 of us researched for 2 weeks and discussed the solution almost everyday. After all, we could't agree on a single result since there was different possible scenarios. Then we explained everything to the one and asked him to choose between our solution.

Bug Ratio and "Blame" tools

It is really important to know that we don't blame people for their code! Because we all know that as humans we all make mistakes and more importantly as a team we have to support each other, not to blame each other, So how can we use the blame tool? (as git called)

Believe it or not, we know each other when we work together for a portion of time and we use this knowledge to understand why someone did something. To be more specific what is your reaction when you have to change an old code and you think it is not efficient, or it is not what you think it has to be?

* If you simply change it to what you think is correct, or you afraid of changing it at all then you need to gain more experience, please don't change anything and ask someone with more experience to work in pair with you for some time.

I normally try to understand the method, what it does, why the one implemented it like that and where it has been used, If I still believe that it is not good and I want to change it, I will check the history line by line using blame tool to understand the process of change and to define the owner. If the code has been changed dramatically by time, or if the owner is someone with normal bug rate, I will refactor the code immediately. But if the code has been written by someone who has low bug rate, (someone who maybe has a clue about the future) I will talk to him first, because it is probable that he knows something that I missed.