Twodee’s kitchen

Contextual exceptions

2019-10-22T00:00:00+00:00

Exceptions are beautiful programming constructs that are often misunderstood. They can allow you to effectively deal with cases where your application may fail AND provide a way to recover from it. Although, many don’t seem to understand why and when to really use them.

Some end up overusing them to an extent that it gets difficult to understand the intent of the program and some (under/ab)use them in all sorts of ways I can’t imagine.

One of the worst ways to deal with exceptions is to simply print the stack trace. Java provides you an easy way to do that with e.printStackTrace(). It’s great for debugging in development, but in production? A complete no-no. Unfortunately, I have seen enough code in my life to believe that people do have the habit of pushing that to production.

Another school of thought is that exceptions are evil and people downright try to avoid them. Yes, they do provide another exit point for our application to use, but that’s only for exceptional circumstances. That means, it’s going to be a rarity, and you should be able to gracefully recover from it without ruining the user experience.

If your application runs into exceptions just as much as a valid flow, you’re not using exceptions correctly, but that discussion is for another day. I am not going to talk about whether exceptions are good or bad and how they are useful or not.

What I really want to talk about is exceptions within contexts.

A common way to handle exceptions is to just rethrow them till the final caller ends up handling it. So when registering a new user, your data mapper throws an exception on persistence:

DataMapper.persist(Object object) throws PersistenceException

And how is it handled?

class UserRepository
{
	public void save(User user) throws PersistenceException
	{
		...
		dataMapper.persist(user);
	}
}

Now when we go higher on the domain layer, we expose the same exception to the caller - the AccountService.

class AccountService
{
	public void registerNewUser(User user, MetaData data)
	{
		try {
			...
			repository.save(user);
		} catch(PersistenceException e) {
			// do something with the exception
		}
	}
}

So your service now has to handle the PersistenceException thrown by the DataMapper.

Wait, do you see what happened there? You just exposed your implementation details to AccountService - which is two levels higher than the data mapper.

The point of abstraction is completely missed at this point. While you were trying to abstract things away using a repository in your service layer which in turn talks to a data mapper, you leaked an implementation detail embedded in your method signature quite explictly.

How should the service layer deal with a implementation specific exception? Fear not, we can do worse.

We can rethrow that and go one layer higher - maybe in a Model-View-Controller design - the call to the Service Layer is made from the controller’s context, so now the controller now has to handle the exception.

Inversion of Control, right?

At this point the controller has no idea why its being handed a persistence exception. If it’s still hard for you empathize with the controller, let the controller rethrow it! Now where does the exception end up? The top level caller - the end user.

The end user now sees a massive stack-trace:

javax.persistence.PersistenceException: Unable to build entity manager factory
at org.hibernate.jpa.HibernatePersistenceProvider.createEntityManagerFactory(HibernatePersistenceProvider.java:81)
at org.hibernate.ejb.HibernatePersistence.createEntityManagerFactory(HibernatePersistence.java:54)
at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:55)
at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:39)
at com.uzh.platform.api.util.AssignmentUtil.findAll(AssignmentUtil.java:20)
at com.uzh.platform.api.services.GetAssignments.getAssignments(GetAssignments.java:22)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:151)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:171)
...

Maybe you have been printing the stack trace all your life and aren’t intimidated by it when you see it. But think about it from your layman user’s perspective. Does it make any sense to the poor user at all?

No. They would panic and call customer support immediately, and you know what they’d do? They would send you a picture of the stack trace clicked on their phones. Which at this point, is essentially rethrowing the exception back to YOU. Only this time, your money is on the line.

Now, it’s your turn to ask them what really happened and what they clicked and how this came about. You try to understand the context and fix the issue.

If you have been developing software for more than a year, you’ll know that when you go higher the abstraction level - the more vague the cause of the problem becomes. Most end users are not going to be of much help unless you have a lot of well paid and patient support representatives.

It could have been avoided by simply saying something sensible like: “Something went wrong in our servers! We are fixing it ASAP.” to the user.

This would make perfect sense to your customers, and in most cases they’d simply wait till it gets fixed. It’s understandable in their context.

I take this opportunity to coin the term “Contextual Exception”.

The idea is to wrap implementation specific exceptions thrown by low level parts of the code to domain specific exceptions that fits the context from which the method was called till the point that it stops being an exceptional case for the caller in the context.

This can later be handled sensibly at a higher level of abstraction to deliver a useful feedback to the caller and continue.

So the PersistenceException thrown by the data mapper in our first example, can be handled by the UserRepository itself, which can then find the root cause of the exception and handle it appropriately. The UserRepository understands what could have happened in the data mapper layer and can deal with it.

For example, in JPA, a foreign key constraint violation throws a PersistenceException as well. You can catch that in your repository and wrap it in a (domain specific) UserExists exception, which makes perfect sense to the AccountService. Now, when registering an account, a non-unique email address may be entered. This is expected by your controller.

But even though you check for uniqueness before hand with a select query, there may be some race conditions when you may end up entering a value which becomes non-unique. Like two users simultaneously trying to register with the same email address. One of them will register first and your check at the beginning will fail.

The only way for JPA to tell you that such an exception occurred is by throwing an exception at a low level. In such a case, your repository has to graciously recover from that and wrap it into something meaningful to the upper layer - that is the account service.

Then what about the exception to be thrown to the controller? In most cases, you shouldn’t throw an exception back to the controller. You should handle it right there in the service layer and casually let the controller know that the email address has already been taken.

Why? Because this is not really an exceptional case for the controller. The controller expects users to enter non-unique email addresses and usernames. It has no idea about the persistence layer or any of the implementation details which may raise race conditions.

The account service may use some sort of notification system to convey the message to the controller and the controller can turn it into a user friendly representation or just pass it to the view to do that. Now the account service can look like this:

class AccountService
{
	public void registerNewUser(User user, MetaData data, Notification note)
	{
		try {
			...
			repository.save(user);
		} catch(UserExists e) {
			...
			note.put("email", "The email has already been taken");
		}
	}
}

class Controller
{
	public void registration(Request request)
	{
		...
		accountService.registerNewUser(user, meta, note);
		if (!note.isEmpty()) {
			// do something useful
		}
	}
}

One thing to note is that exceptions can be costly, especially while catching and unwraping them. That’s one reason why people call them evil. But exceptions are there for a good reason - To deal with exceptional cases, and such cases shouldn’t happen in your normal flow of operations, at least in that context.

A user entering non-unique email addresses is not exceptional to the controller, but a sudden connection loss with the account service on a network is very much an exceptional situation.

In that context, the appropriate party should raise an exception and this should be handled by the controller in a graceful way while logging the cause for debugging.

Yes, there are some cases where rethrowing the exceptions and letting them bubble up is the best option but that’s usually when that’s your only option. Again, it largely depends on the context and how your system is built.

If you are going to (re)throw exceptions, ask yourself this - does it make sense for the one receiving the exception? In most cases, your answer would be “no”.

Handle exceptions with care.

Design patterns - to do or not to?

2019-10-19T00:00:00+00:00

When I came across design patterns a few years back, they seemed like a solution to everything. There wasn’t a problem that one of the design patterns couldn’t solve. It was almost analogous to mobile apps. There’s an app for everything these days.

For those who don’t know yet, design patterns are well defined patterns of reusable solutions that are followed to solve commonly recurring problems in the software engineering world.

The term design patterns surfaced from the legendary book by the Gang of Four (Erich Gamma, et al.) - Design Patterns: Elements of Reusable Object-Oriented Software (1994). More than 2 decades old, this book is still very much relevant in the fast-moving software industry.

Software Engineering being a young profession, we are struggling to find standard ways to do things. While we are trying to build something big, we end up adding dirty fixes and hacky features. No two engineers are same, and to achieve a common ground organizations enforce conventions and patterns to adhere to while individuals contribute.

My first encounter with design patterns was when I needed to establish a way to keep a particularly expensive database connection to just have a single instance to be accessed everywhere. That’s when I came across the Singleton pattern.

I looked around for implementations of the Singleton pattern, and most examples I came across had a structure similar to this:

public final class HeavyObject 
{
    private static final HeavyObject instance = null;
    private HeavyObject() {}

    public static HeavyObject getInstance() 
    {
    	if (instance == null)
    		instance = new HeavyObject();
        return instance;
    }
}

There are a few things to note here:

The constructor is private. So, if you tried to instantiate it by calling: new HeavyObject() it wouldn’t work.
The HeavyObject instance is a static property. Which means global state.
You can only access it by calling HeavyObject.getInstance(), and it will return the same instance everytime.

Perfect!, I thought, as I replaced my current database connection with this, plus I enjoyed the lazy initialization goodness. It worked great and served my purpose. At some point, I was inclined to using it for almost every problem that I came across, or at least think about using it.

Turns out, global state isn’t a particularly great idea. Global variables can wreak havoc if not used judiciously, and you may have tightly coupled pieces spilled everywhere in your project.

Enter Dependency Injection: Trying to find ways to get around the Singleton pattern without using static, I found Dependency Injection (and the concept of Inversion of Control). In simple terms, you don’t create the dependencies you need, you ask for it and someone hands it to you.

Anthony Ferrara, in his video presentation about DI explains it with the analogy of a house building robot. Your house building robot is programmed to take blocks of lumber and fit them together to build the walls. When it needs to build a doorway it has two options - To create each door out of raw materials, or to take a ready-made door and just add it to the house.

The door is a dependency for the robot to build the house. If it were to create a door everytime it had to add a doorway, and then in the future, you realized that you don’t like the doors that the robot built - you’ll have break that door, everywhere.

public final class HouseBuilder 
{
    public HouseBuilder()
    {
    	this.block = new Block();
    	this.door = new Door();
    }

    public void buildHouse()
    {
    	makeWalls(block);
    	makeDoorWay(door);
    }
}

This class knows too much about how doors and blocks are built. Don’t let your classes know too much.

Dependency injection solves the problem by handing over a readymade door to the robot. Now if you’d like a different door, you simply swap the old door with a new door.

This decouples your wall from any specific implementation of the door. The robot just asks for a door that satisfies specific needs like the dimensions in the form of a contract.

Using dependency injection helps reduce tight coupling of code and enables us to switch between implementations of interfaces. This is inversion of control. The caller feeds the dependencies to the callee, rather than the callee creating its own dependencies.

So we can change it to something like this:

public interface Door
{
	public void lock();
	public void open();
}
public final class HouseBuilder 
{
    public HouseBuilder(LumberBlock block, Door door)
    {
    	this.block = block;
    	this.door = door;
    }

    public void buildHouse()
    {
    	makeWalls(block);
    	makeDoorWay(door);
    }
}

Now that the door is an interface, we can switch out implementations in anyway we like. We may want a glass door at a store front and a wooden door in a house, and we can do that by simply injecting a different implementation.

Dependency Injection is often carried out by Dependency Injection Contrainers (DIC) which create the dependency for you, usually utilizing reflection to traverse multiple levels of dependencies. This is often useful for managing dependencies but not always necessary.

DICs can easily help you replace the static-based implementation of Singleton by asking the DIC to supply the same instance of the dependency. This easily reduces tight complexity and the evilness of singletons and global state. So this is how I ended up solving my Heavy object Singleton problem.

So far, we have established the advantage of writing to interfaces than to concrete implementations, we can also take advantage of another pattern called Factory pattern. Dependency construction isn’t always an one liner new LumberBlock(). Factories are methods that enable creation of dependencies with complex instantiation.

This can be further extended by using the Abstract Factory pattern that creates an object depending on the context.

For example, if you’re making a store, you’d probably want to be supplied with glass doors. When you are creating a fort, you’d probably looking at strong iron gates. The abstract factory deduces that type of door to be supplied depending on the context.

As it is evident from the discussion above, it’s very easy to get lost in patterns while talking about them. You start off with some pattern and end up talking about something else.

Usage of design patterns has been growing gradually, and with that comes individuals abusing them to an extent that it exponentially deviates from the original intent. I’ve come across a lot of codebases that agressively use patterns and even anti-patterns like Singleton.

The concept is tempting and as a consequence codebases turn into a huge mess of design patterns on every page. I am guilty of committing this mistake.

What I have learned from experience is to let design patterns surface based on a need rather than a want. Like on my first introduction to Singleton, I needed a way to access just one instance of a database connection.

When you start wanting to use patterns for the sake of using patterns, that’s when your codebase starts smelling and it ends up looking like:

JSONDataStorageStrategy datastore = AbstractDependencyBuilderFactory.createSingleInstance("jsonDataStorage");

It doesn’t make any sense (not at least to me), and soon gets hard to decipher over time (I guess that’s where Java gets its bad rap from).

Design Patterns isn’t a silver bullet - they have their usecases, pros and pitfalls. They come with their own gotchas and they fit in some cases better than others. Just because your colleague used a facade over the data mapper doesn’t mean that you’ll have to. Many libraries actively incorporate some bad design techniques this way.

For example, Active Record is often considered an anti-pattern is being used at many popular ORM-libraries with big corporations backing them.

Anyhow, design patterns can be a great thing if used judiciously. Deciding on a use-case and trying out a simpler solution first should be done before jumping onto the GoF book. This can yield great results and improve the codebase signficantly.

Recently, I came across the Notification Pattern from Martin Fowler’s Further Enterprise Application Architecture development. This turned out to be a great alternative to my previous exception-based validation approach by saving me some computational cost and allowing me to aggregate all the errors at once.

With that said, we invent new patterns everyday and sometimes they add additional complexity to our codebase. One good rule of thumb while writing code is to make them reusable and do just one thing.

One of the best strategies I have devised is to make a very local solution to a local problem first, and then generify it along the way.

The entire goal of design patterns is reusability, and they may have consequences and tradeoffs that come along with them which we have to deal with.

Questioning whether it’s really worth the additional complexity and designing a mitigation strategy for dealing with the tradeoffs beforehand will take you a long way than simply using a pattern just because someone else is.

You ain’t gonna need it.

Another Pycon, more community

2019-10-14T00:00:00+00:00

As I fly back to my university from Chennai after Pycon, I cannot stop recalling everything that has happened to me for the last 3 days. Leaving the Airbnb and going towards the airport wasn’t an easy thing for the mind. I had to pen down what’s been going on in my head for a release.

After finishing my exams midst Durga Puja (which I was really looking forward to), I spent a few gloomy days in the hostel because I couldn’t go back home. Desperately needing a break, I was counting my days till Pycon.

To be honest, I wasn’t as excited I was last year to go to Pycon - partly because I just had my exams done, and I couldn’t enjoy the 3-day holidays I got after that.

I packed all my essentials on my backpack and ran towards the airport thinking that this will suffice. Boy, was I wrong. Upon reaching Chennai, I met up with Kuntal (@hellozee) and a two more #dgplug members with whom I will have been staying for the next few days.

We got into the Airbnb, me being socially awkward, stayed silent for a while. The rest of the #dgplug gang joined in with Pizzas and Pepsis. I got my mental cooldown over a game of Uno and having pizza for the first time after Fall, 2017.

“Dedipyaman” is not an easy name to pronounce. We had a good laugh over people trying to pronounce my name. Even the Bengalis in the group were struggling. Now you know why this blog is called “Twodee’s kitchen” and not “Dedipyaman’s”.

Well, I was living with developers from different parts of the country. We had met for the first time, this did feel like home.

At the Chennai Trade Centre, I made sure I collected my stickers first. I participated in my first challenge at the Jetbrains booth to bring home some goodies. I didn’t want to miss out on a T-shirt featuring my favorite set of IDEs in the world.

Thoughtworks (I know it as the company Martin Fowler works at) gave me a nice Wireless Charger for a simple Binary Search snippet re-arrangement. I don’t have a phone that supports wireless charging. If the generous reader would like to gift this poor soul with a phone that supports Wireless charging, they can have a virtual hug from me.

AQR presented me with a T-shirt, a cool coaster and a mousepad with USB slots. I am yet to figure out how that works, but as the first person who solved your Pycon challenge - I thank you for the goodies.

As for the talks, I wasn’t sure how the silent conferences would turn out to be - but for a change, it worked out pretty well!

I really loved David Beazely’s (@dabeaz) concluding Keynote (although I’m not sure what his original intended topic was). He wrote a stack machine in front of a live audience, and then went on to do WASM with Python while making mistakes - like a normal human would. It was really fun to listen to his insights, and he deserved the whole standing ovation at the end. Python’s (or any language for that matter) future is what we make it.

Pycon is not Pycon unless you walk till you tire your legs out. I went to random people, said hi, talked about our experiences. Met a couple of old faces too. Interestingly, a significant portion of Pycon attendees don’t write Python. Well, good to know that I am not the absolute minority.

It was all cool and fun. At some point the goodies got heavy and it was a challenge for me to carry my backpack. Two days back, I had squatted and deadlifted after a long three week break. The DOMS (Delayed Onset Muscle Soreness) didn’t fade away, and the backpack somehow equalized the pain by hurting my upper back while the DOMS killed my lower back.

I had a backpack situation that the genius of @hellozee helped me solve. Lesson learned: Carry another bag next time.

But this time, Pycon for me was not really about the conference - it was about the people. I realized, you don’t really feel the impact of Open Source at a personal level, unless you spend time with the community in real life. Yeah, Open source is changing the world - but how its bonding people from all groups, ages, sexes, races - can only be felt when you play a game of cards or have dinner together.

Or when you have the tech-stack wars: No one writes JS willingly, Java yields loooong lines…

Unfortunately, I live in a tightly packed bubble with almost no exposure to the outside world. The only communication I do get with like minded people is through online channels. I didn’t know about #dgplug being this online IRC-based thing and meeting at least once a year at Pycon.

The way they came together as a family, laughing at each others jokes and enjoying the moment - it was something special. I have had the chance to meet and talk to them about what they do, what tools they use on a daily basis, their adventures in life. A real life human-to-human experience sharing.

So, here’s my take on my experience in this year’s Pycon:

Coding solo is great, you can write code all the day alone and get things done. The community is greater, you can empathize, share, and grow together.

I would like to thank all of the #dgplug folks to make me feel welcome and a part of the group. It was something I really needed, but didn’t know that I did.

As I arrived at my dorm room, I conclude this with a heavy heart. I look forward to the next conference and hopefully will be a more social person.

Pycon was all worth it.

If you haven’t read it yet, check out this post about my first time Pycon experience: How Pycon India 2018 changed the way I see things. To sum up, I can reuse what I said on my last Pycon post:

It was never just about Python. It has always been about the community.

PHP sucks, can it suck less?

2019-09-26T00:00:00+00:00

PHP has garnered a bad reputation in the software market. Many developers hold strong opinions against the language and to some extent, that’s correct. Over the last few years, although PHP has gone through an evolution and is it the same “Fractal of Bad Design”?

Yes, I admit it. PHP sucks.

And I shamelessly write PHP code, so I must suck too, right?

As popularly stated in the famous article PHP: a fractal of bad design :

PHP is an embarrassment, a blight upon my craft. It’s so broken, but so lauded by every empowered amateur who’s yet to learn anything else, as to be maddening. It has paltry few redeeming qualities and I would prefer to forget it exists at all.

The article has only gotten popular over the years, and it’s still being circulated over Quora (One of them being the founder himself, Adam D’Angelo).

And while the article came out, yes, it was right. It was badly designed and badly implemented. The users made it worse. While PHP came along, it didn’t originally plan to be as massively used as it is today. Since it was so easy to adapt to, people started using it everywhere. It worked to some extent.

As soon as the web got more popular, we changed, our needs changed. PHP didn’t. It was still stuck behind, probably due to the community? It was inconsistent with its naming, it had the insecure mysql_* functions built into its design. There were a lot of gotchas in the language, and having used PHP for a long time, I know that it is a pain.

But this is 2019. The article was written in 2012. I am surprised people still keep quoting that article everywhere!

C’mon people, PHP has changed (evolved) a lot, don’t tell me the public eye is too blind to see it.

PHP has had some major pushes like the HipHop to HHVM movement from Facebook and PHP 7. Developers have recognized the issues that came along with it, and they have been addressing it so far. If you are living in 2019 and still writing mysql_* functions (or blaming PHP for having that), you seriously need to learn to RTFM.

So why this hatred still?

Well, as long as something’s popular - people will hate it. People hate Java, people hate C++. When millions use your product, you cannot expect everyone to be happy customers with every design decisions that you make. There will be people who don’t like your approach and that’s true for any remotely popular language.

People often compare Python to PHP in the web context. I have nothing against Python, I think it’s a great language that fits the purposes it was intended for (scripting?) and purposes it was adapted popularly for (AI/ML/Data Science?).

But here’s where Python fails to impress me against PHP:

It’s slow. Not a deal-breaker (especially on the web), but I am making arguments for the sake of making arguments. If you still complain about PHP being ugly, I can complain it about being slow.
It needs a framework for anything web. Initially, when I just wanted to get a Python application up and running for the web, I had the community continuously push me over to use Django or Flask. I hate being coupled to a framework, as many others would (and should) be too.
Whitespacing? Not a fan. Again, it’s not a deal-breaker, but having whitespace mean something doesn’t make sense to me. I understand that it was a design decision to keep the lines cleaner, but when things break because I missed an invisible whitespace - it hurts my feelings.
Its Object-oriented model is alien to me. Access specifiers are done by enforcing conventions with underscores? Okay. No. Maybe it works for some folks, but I like things being implied explicitly rather than implicitly.

But okay, Python is a great language. It works great, it’s got a great community and yet - there are people who hate it. And that’s okay if you are remotely popular anywhere - you will have people not liking you.

Javascript on the other hand - It’s something I really don’t like. It’s a matter of personal opinion. Especially after the fact that some Javascript dudes who are a few years older than me were trying to shove Node.js down my throat and bashing PHP for the time I was in front of them. Asserting that Node is far superior, safe and faster than PHP (and anything else for the web) and I should learn Node right away.

They went as far as saying that PHP invented SQL injection. I stopped trying to speak at that point.

Coming back to PHP:

Do you still have/write legacy PHP code that follows the arcane PHP 5 approach?

I have a list for you:

Start writing OOP

While you can still write procedural PHP, the community has moved towards an object-oriented approach. It simply fits the new model and works great to have you structure the code well. With object-oriented, several clean coding practices like SOLID and DRY are automatically implied.

OOP silently enforces clean structuring of your codebase and keep things separated better. Of course, it’s optional, if you like writing spaghetti code, no one’s stopping you. You can make the worst out of PHP and give yourself a bad name. But that’s entirely up to you, any language will allow you to do that. Not just PHP.

Strict type as much as you can

While we are at the subject, also use Strict Types. It’s as simple as: <?php declare(strict_types=1) at the beginning. Static typing surely helps you keep things consistent and safe. Yes, PHP is a dynamically typed language, and we need to squeeze that feature out of PHP sometimes. But in most cases, going by the safer path with strict types can save you from a lot of weirdness and unpredictability at runtime.

Namespaces, please

The include statements on top of the page are no more common and the community recommends that you use namespaces to “import” modules you need to “use”. Its a means of abstraction over your raw PHP files that allows you to encapsulate the include logic.

Yes, it could be weird to use \ as the namespace separator at first, but you’ll get used to it. Get rid of those includes and start using namespaces to put things into their correct places. Which brings me to my next point:

Composer

If you are planning to start a PHP project, get composer immediately. It’s a dependency management tool which allows you to define your dependencies, your application and test entry points and load dependencies from the central Packagist repository as you need them. It generates an autoloader for you, and that’s the only thing you should include in your entire project.

Throw away `mysql_*`

All the mysql_* functions have been deprecated for a long, long time and it has been removed in PHP 7 for good. So if you are still complaining about mysql_* functions being bad, please upgrade your PHP version. The best way to deal with a database as of now is to use PDO with prepared statements. It’s a generic API that works quite well with a vast array of databases.

The things that I like about PDO are: it’s clean, relatively modern design, object-oriented and consistent. You will move to exclusively use PDO in no time once you start a project with PDO.

Again, don’t create DB wrappers like DBConnection extends PDO. Just don’t. If you need some sort of abstraction over PDO, check out the Data Mapper pattern and some ORM like Doctrine (and not some evil Active Record variant).

Separate your concerns

Most of the bashing PHP gets today is because new developers mess it up so bad, that it smells worse than Javascript (I am opinionated, sorry). Because its easy to learn and get started with, newbies just can’t resist themselves from writing hacky code and deploying it to production.

Other languages don’t get this because

a) they have a steeper learning curve

b) they give out a strict design strategy beforehand.

Fix this by separating your concerns. I remember once I used to copy-paste portions of long functions to other files to do the same thing, but slightly differently. I understand why beginners do that.

Start off by making your functions smaller.

Break your codebase down to small pieces acting independently doing exactly one thing. Read more on SOLID and DRY principles.

If you have a User class, don’t allow it to be able to create a message, encrypt the message and also send the message to another user.

Incorporate libraries from the internet, people solved your problems with better testing well before you did. And they did it better. Whilst we may be tempted to have everything custom made, we tend to deviate from the actual business needs. Focus on your business logic, use what’s already available.

But no tight coupling.

PSRs to the rescue

And finally, follow coding conventions and read on PHP-FIG. The PHP Standards Recommendations (PSRs) will allow you to have a consistent codebase that others can easily understand and extend. The standards will help you to write code that can be compatible with other code written by others, and that will save you from cursing yourself 3 months after you write some bad PHP code.

This recommendation applies to every language in general. Follow coding conventions and strive to write better code. Of course, no one can stop you from being a “code-rebel”. We don’t have the technology to stop you from that yet.

(Yes we do, coding standards checks during integrations can keep idiots at bay)

A few concluding words

Yes, PHP sucks. So does every other language. You just gotta deal with it.

It’s up to you and your team to write code that looks like poetry rather than an ugly war cry. You can write severely bad Java code even with its verbosity and static typing.

You were scared about starting your next project with PHP because your coworkers will judge? Go right ahead and do that. It’s your job and it’s your tool.

Your coworkers are probably too free, maybe their code is still compiling.

Technical debt

2019-09-23T00:00:00+00:00

Software development is cursed with an implied cost at any stage, no matter how elegant your solution looks at any point of time. The cost of fixing what you messed up. Humans write code, and code is prone to ugliness. This piles up over time and dealing with it is a challenge every team faces many times in their lifetime.

While building any sensible system with code filling more than a page you start breaking things down to pieces, if you aren’t you should. When you start abstracting things away into separate smaller modules, you start thinking about how to put them together. Oftentimes, you need to add one tiny feature and then one more, while trying to keep the initial structure you had in mind intact.

Although the idea was clear at the beginning, the pieces don’t simply fit together as easily as you’d want them to be. So, you mess around the structure of the codebase a bit, just fine little harmless tweaks. As features accumulate, you start to see a big change that module’s structure.

What starts off as a clean, well-structured module now became this horrible mess of hacky additions of features. At some point, you stop understanding what it was supposed to do, because it’s doing too many things.

But it works.

So you leave it behind, put a checkbox in your TO-DO list and move on with the next module. Then days after, your requirements change and you need to modify that module once again. Or maybe you’d like to integrate it with some other module. The confusing module is now hard to read, understand and extend. Moving any further would be an impossible task with it.

You’ve two choices at this point-

Discard it entirely and start fresh
Fix it

Unless you’re building your first application which doesn’t do much, the first is usually not the best option. Especially in a project where you put significant amount of time thinking about and developing that module. It works, and it has some clever crafted code inside it, it’s just turned messy.

Now you have to take a step and fix it.

This is Technical Debt. As in monetary debt, if you don’t repay your dues early on, the interests will accumulate and push you into a hard-to-return-from point.

When you thought, “hey, this works for now, we can fix that later” you fell a little deeper into the debt trap. Technical debt is the additional cost you implicitly added to your project by choosing an easier path that you will have to pay for in the future.

Now what would have taken you 3 days of work, will take you about a week. It’s the cost of fixing the code (paying off the debt) + the cost of implementing the new feature.

How do I prevent technical debt?

You don’t. You can only minimize it. Think of it as a financial system, and do exactly what you’d do when you loan money off your bank. You pay the dues gradually. Once you are done implementing the first feature in your module, you take some extra time to remove that mess and make it saner.

Yes, you did put extra time into it, but it lessened the burden for the next feature and decreased the overall cost. This gradual removal of mess enables you to fix the parts of your code that needs the most modifications over time, and hence needs to be the cleanest.

Cleaning it off gradually over time will make sure the debt doesn’t accumulate to the point of no return over time. And it helps you keep the internal code quality fresh at the points where you need it the most.

Recently, in a system I had used a factory to convert transfer objects to domain entities. It was initially meant to do just that, but I ended up validating the data and checking for redundancy from the persistence layer inside it. All within a factory. It was too smart at that point.

Eventually I kept using this factory in too many pieces, and adding more logic to make it smarter. I started having a hard time connecting the pieces together and that was a clear violation of SRP. I had to take a day off just to dumb the factory down to just be limited to creating the objects and not do too much.

Tight coupling is a major evil in a modular system and can often lead to code smell even in the most initially well thought out business logic. Coupling can end up contributing to the technical debt you accumulate over time, and you simply would be confused when something stops working in the cleaning up process.

When you start paying off the dues at a much later stage in a tightly coupled system, you find yourself breaking things even more than you’d do normally. At that point the paying off process becomes a nightmare and takes huge effort to fix.

Unpopular opinion: Don’t fix something that works and you seldom change.

If your co-worker built a module that works perfectly fine and integrates just well enough for you to go on with - don’t bother about fixing it if you don’t modify it frequently enough. There’s little reason to fix something that confusing and doesn’t need fiddling with. If it meets your business requirements, leave it alone.

Finally, write more tests. Sometimes, we just need to add a quick fix, inject a hacky piece of code to make something work or to slightly change its behavior. What we don’t see at the first glance are the side effects. We tend to look more at whether that quick fix solved the actual problem, and we miss out the side effects it brought along with it. Tests help out with that, you get instant feedback.

That’s why I am an advocate of Test Driven Development. It gets you to move those sticky pieces long before you even are done implementing a feature, and hence reduce the cost. A slight cost added is the cost of writing those tests, but you would have to write them anyway.

Technical debt is inevitable in software development, and it will come around. And sometimes we need to trade off elegance for making things work, that being said, paying them off sooner will make your life a lot easier.

Don’t fail silently, scream!

2019-09-20T00:00:00+00:00

Your business logic most certainly has the error, not someone else’s library. Use error messages, your best friend.

While working on a recent project, while I was making remote calls from a client application to a server running on the JVM everything worked until after certain number of requests to the server, the application kept failing.

The server got irresponsive after too many requests, irrespective of the frequency. It just kept stopping.

I was presented with a very specific and helpful message: TSocket: timed out reading 4 bytes from {hostname}:{port}

Initially I thought I may have a very short timeout indeed, so I just bumped up the read and write timeouts to 60 seconds to see if it made any changes. It didn’t. It kept stopping after a certain number of simultaneous requests, and it wouldn’t start until I restarted the server.

Something didn’t make sense, I had tested every edge case of every unit properly - yet while trying to integrate it with the rest of the modules, something simply stopped working.

I kept looking at the client side, and never bothered to look at the server side - because everything seemed “just fine”. Needless to say, I failed and failed hopelessly for a week.

I ended up asking the mailing list, opening a Jira ticket with a critical bug report and had long discussions with developers and users. I was surprised no one had this issue, and they couldn’t come up with a proper answer to this.

Let’s just say the community support isn’t that great for this specific framework.

After a week of forgetting it, working on the business logic, writing more tests I finally decided to fix this issue. I kept reproducing the bug. Then, I noticed a pattern.

It wasn’t just a random number of requests - it was exactly 10 requests. It kept failing after 10 requests, which was odd. Any server should be able to handle 10 requests without any sweat. And why 10? It seemed too human-istic.

For a human being to choose the number 10 (in decimal), very common. For a computer? Not so much.

Maybe I wrote a magic number 10 somewhere? Probably.

Then I started checking out the logs my java server was appending, and the persistence layer likes egesting the SQL queries being executed to the logs for some reason. Which ended up being very useful.

For every request that I made, 2 queries were being executed. Wrongly, they shouldn’t be executed in the first place (which really wasn’t the cause of the problem, but it did help me fix the problem).

So, 2 queries, multiplied by the 10 requests gave me 20 queries in total. That number seemed familiar.

I started checking the connections in the server, if I had left anything open. Nope, every socket connection was closed properly.

Then maybe the database was blocking my connections? 20 queries are too much? Is there a MAX_CONNECTIONS limit somewhere? I was pretty sure I didn’t specify a limit like that anywhere, yet. It’s my development machine.

I checked out my MySQL server with the handy command:

mysql> show processlist;
+-----+------+-----------------+---------------------+---------+------+----------+------------------+
| Id  | User | Host            | db                  | Command | Time | State    | Info             |
+-----+------+-----------------+---------------------+---------+------+----------+------------------+
| 831 | root | localhost       | NULL                | Query   |    0 | starting | show processlist |
| 847 | root | localhost:34782 | Application Service | Sleep   |   54 |          | NULL             |
| 848 | root | localhost:34784 | Application Service | Sleep   |   54 |          | NULL             |
| 849 | root | localhost:34786 | Application Service | Sleep   |    2 |          | NULL             |
| 850 | root | localhost:34788 | Application Service | Sleep   |   54 |          | NULL             |
| 851 | root | localhost:34790 | Application Service | Sleep   |   54 |          | NULL             |
| 852 | root | localhost:34844 | Application Service | Sleep   |   52 |          | NULL             |

AH-HA!

My connection pooler has been spawning way too many processes.

Then I went to my persistence config file specifically looking for my connection pooling config. And here it was - maximum pool size was set to, guess what? 20. All the dots fell into the their correct places, they finally connected.

My connection pooling agent was spawning (or leasing) a new connection for every query I made - and that was expected. What was not expected was that the connection pooler never released them.

Then, I finally looked at my business logic. And guess what? All that pain for a week was for a silly, silly error.

I didn’t commit or rollback the transactions I started. Uncommitted transactions don’t leave the pool, and hence for every request- I had 2 connections.

As soon as I reached 20 connections, things broke. Fixing the transaction worked like a charm.

Lesson Learned:

Close your connections.
Commit your transactions.
Your business logic has the error, not the library.

Here’s where error messages could have been super helpful.

Thrift doesn’t like to carry forward the exceptions thrown by the server to the client which it doesn’t know about.

You have to explicitly tell it, “hey, these are the exceptions that might occur if something goes wrong. Ask the client to handle them”. And if you don’t, well it throws a generic exception with generic messages. Not very helpful.

When I check the server console, when it’s being run with Thrift, it doesn’t fail with an exception either! Java is usually super verbose, but in this case - it failed silently. Surprising.

If I hadn’t run it by the server remotely, but with a custom caller within the server, maybe it would have failed while screaming something like “MaxConnectionsReachedException” (I don’t know what it really throws) with an elaborate stack trace. Now, that would have easily led me to the path I really needed to walk on.

That’s why exceptions are your best friends. Just a simple error message or a stack trace could have pointed me to exactly what went wrong, but it simply wasn’t visible to me.

Well, this is one of the shortcomings of my RPC framework. There are downsides to everything, but it taught me a valuable lesson:

In a development environment, always fail while screaming. NEVER fail silently.

Also, I need to look at my business logic more and integration test more often. And you should too. Stop blaming.

Dedipyaman

Learn Programming The Right Way

2018-11-27T00:00:00+00:00

A recent train journey inspired this blog post. As promised, albeit being late, this is the blog post on how to start programming if you are a complete beginner.

This is a continuation of my post: Learn Programming in 24 Hours!

Now here are a few gotchas about this post-

I do not claim to be an expert in the field.
It is not a silver bullet or a magical todo list to take you from zero to hero in less than X time.
This is not a tutorial of any kind. This post just gives a few pointers to the absolute beginner so that they can get started a bit more easily.
This post doesn’t guarantee success. This is a compilation of my experience.

Find your motivation

So when I wrote my first line of code back in 2010, I was a 10-year-old sixth-grader. I was impressed with the game Smackdown Vs Raw 2010 by WWE and THQ, especially the “Create a Wrestler” feature - which allowed players to create a fictional superstar with appearance, attributes, and moves of their liking. I wished that feature to be available in other games as well and was curious about how it was done.

I wanted my own storyline and characters.

So, I decided to figure it out. I grabbed my old laptop, googled “How do I make a video game?” and started hunting down ways to start making a game and ended up finding a tool called Game Maker which would allow you to create tiny little 2D video games. That was my introduction to game building. That was a good stepping stone, but I still didn’t know how to make a game with an actual storyline.

I wanted control. I read more and more and found out that games need to be “programmed”. I envisioned a GTA-like open-world game with my own characters, storyline and weapons. And to do that I had to learn how to program.

Enter QBASIC - my first programming language. This was being taught in my class, but I never cared for it before. Now I did. Because it meant something to me now. This is a tool I can move forward with.

I started with the infamous “Hello world” program and moved on to write programs emulating a simple Quiz Game with menus and everything on the Command Line.

This led me to stick in and never move away from this craft. I haven’t build a game yet, but I have moved in different directions and I am grateful to my 10 year old self for that.

So here’s the bottomline:

If you don’t care about programming, you are never going to learn to program. Programming itself isn’t fun. Finding ways to solve a problem is fun. It’s the same thing as solving puzzles. No one likes adding numbers, yet people like solving Sudoku puzzles. It’s a challenge you set up for yourself. Numbers and addition is just the tool to solve that.

So find yourself a problem, a motivation if you will - and start figuring out ways to solve it. One step at a time. You aren’t going to build a microservice application in one day. Start with smaller ones, make mistakes, learn from them and make a bigger one the next time.

Stop reading! Start writing iteratively, learn from your experience

As wrong that may sound, you learn to program when you program. It’s a recursive process. I have tried sitting in front of a video tutorial or reading books for hours to end up forgetting in a week or so. Reading is practically useless if you are not using it.

I learn more when I write, and you should too. People call it practice. When you start writing code and compile and see it run - you are in debug mode. You try to find the flaws in your program rather than trying to remember what you did. Your brain saves that information for you.

This is how it roughly works. Imagine learning to program for the first time in C. Here’s your first piece of code that your remember seeing on a tutorial:

int main() {
    printf("Hello, world!");
}

You compile it and you are presented with this:

hello.c: In function ‘main’:
hello.c:2:3: warning: implicit declaration of function ‘printf’ [-Wimplicit-function-declaration]
   printf("Hello, world!");
   ^~~~~~
hello.c:2:3: warning: incompatible implicit declaration of built-in function ‘printf’
hello.c:2:3: note: include ‘<stdio.h>’ or provide a declaration of ‘printf’

Woah, to a beginner that’s probably very intimidating. But, if you can read english, it’s not too hard to decipher. Our brain just likes skipping over things.

What it really says is:

In function “main”, printf has been implicitly declared. The compiler doesn’t know where printf is defined. So, it’s asking you to either create a function called printf and try compiling it again OR include <stdio.h>.

Then you recall: Hey, I think I read about that stdio thing, maybe just include it?

You google: “How do I include in a C program?". Some angel on Stackoverflow or some similar thread has already answered this question. And you weren't the first one to come across this!

Then you add this line: #include <stdio.h> to your program. It looks somewhat like this:

#include <stdio.h>

int main() {
    printf("Hello, world!");
}

It compiles successfully and you run ./a.out and it works! Almost…

username@pc: gcc hello.c
username@pc: ./a.out
Hello, world!username@pc

Well, we didn’t want to print the username@pc on the same line, did we? What could we be missing?

It turns out, your code simply printed the text and the prompt of your shell decided to continue from that line. We need a way to print <RETURN> or a new line so that the cursor goes to the next line.

After some research on how to move the cursor to the next line in C, you add the “\n” character to the end of the print statement.

#include <stdio.h>

int main() {
    printf("Hello, world!\n");
}

And you compile it once again, no errors. And you get the output:

username@pc: ./a.out
Hello, world!
username@pc:

It finally works. Your brain remembered and stored it in your “Experiences” folder. Now when you write a similar program next week, you’ll immediately recall that you need to include <stdio.h> and add that \n character.

Remembering pieces of relatable experiences and failures is easier than remembering this chunk of code:

#include <stdio.h>

int main() {
    printf("Hello, world!\n");
}

Try it out, it will almost certainly make a huge difference.

Add development tools to your arsenal and make life easier

While people generally recommend you to use nothing but a simple text editor at the beginning of your programming journey, however software development is never about just one page written in one language that’s compiled by one compiler or interpreter.

Get to know your tools and use them effectively. One absolute improvement in any project is the addition of version control. Pick one, git is a popular choice.

Choose a proper text editor/IDE that integrates well with your programming environment. What works for me shouldn’t necessarily work for you. What works for Java shouldn’t necessarily be great for C++.

I personally prefer Emacs as my default text editor to edit configuration, write scripts, and C applications. Because of it being lightweight, fast and powerful I am more productive with it while writing command line apps.

For heavyweight applications, like the ones written in Java, I prefer IntelliJ IDEA - one of the best IDEs out there. Due to Java’s verbosity, it’s often common to miss out a lot of things. IntelliJ is smart enough to keep recommending improvements and finding issues with your code even before you hit compile. It integrates well with Java build, testing and dependency management tools and makes java development a cakewalk.

Then come the build tools. Choose a popularly supported build tool for your language and use that. You don’t want to manually compile every file to turn them into one final executable. Let your build tool automate and take care of that.

Get a proper dependency management tool as well. Oftentimes build tools come with built in dependency management.

As you move forward you’ll come across tools to unit test your application, find code coverage, add integration testing to it. All of these add to your skillset and help you write better software over time. If you don’t want your code to misbehave, you need ways to build and test them properly before you give it out to your clients.

Listen to your tools

As with the last example, the way we solved the problem was by reading the error message and understanding what it meant and acting on it. What most beginners miss out on is reading error messages.

Almost every newbie who comes up to me with an issue with their code says that they are getting “an error” in their code. When I ask them what the error is, they are either blank or can’t recall what it actually meant. It’s understandable, error messages from compilers aren’t always user friendly, and a lot of experienced developers face this problem too. But in most cases, at least in the beginning stages, the compilers would throw in enough pointers for you to figure out what you are doing wrong.

You just need to know where to look. And that comes with experience. Compilers often print a (un)helpful message along with line numbers and function names to enable you to debug the code. One way to go about it is to read the error message line by line and interpret what that means. The language is often not aimed towards beginners, so you’ll probably need to consult a thesaurus or a dictionary if English isn’t your first language.

Once you have interpreted the error into a simpler version, try finding those line numbers and function names where your compiler/interpreter is pointing you to.

Then, start discarding the information you don’t need. Java often produces a stack trace with a lot of lines describing all the classes where the function was called and declared from. Discard the error messages which originated from the library or function that you didn’t write and ask yourself:

“What did I write that caused this problem in the other library that I didn’t write?”

Then read the filtered error message once again, and you’ll start having an idea on how to go about solving the issue. If you’ve been scratching your head for too long, then you may go ahead and google that filtered error message. Chances are, someone already had that issue.

Being able to read error messages and interpreting its meaning is as important as being able to expressing your thoughts in your code.

Your code is you trying to tell the computer what to do. The error message is what the computer telling you that it can’t do what you asked it to do, because you did something wrong or it doesn’t understand how to do it. This more you improve working with this feedback mechanism, the better you get.

You need to be able to talk to your computer very clearly to be able to write applications that solve your problem.

Programming is a two way communication.

Write programs, a lot of them!

Not writing code is by far one of the biggest problem with students I have met in my university life. Students learn a new language, the grammar and the semantics and just move on with the next cool thing they find without actually trying to stick with one thing and making something useful. They put it in their resume and move on with their lives.

The only way to get better at writing programs, is to write more programs that solve different kinds of problems. Doing the same thing over and over again isn’t going to make much of a difference. You need to find new challenges and solve them every day. One good way to keep track of it is using Github contributions. Looking at the green boxes will help you understand if you are actually improving or not.

The greener your page is, the more you have been improving. If it’s been greying out, this is your wake up call. Yes, there are days when you don’t feel like it. And it’s fine to take a break every once in a while.

On average, I write and delete about 150-300 lines of meaningful code everyd ay. When I say meaningful, it means that the code actually does something useful and not just for the sake of keeping my Github profile green.

If there’s one thing that you can to pick from this list, this is the one.

Make projects! Stop watching those video tutorials about that fancy new technology. Make something first and keep improving on it.

I have come across individuals who have been “course hopping” from one course to another. One minute they learn python, the other minute they learn Android. And I am yet to see anything useful come out of them. Don’t be a course hopper. If you can’t commit to a particular online course (I can’t, I am too impatient), just work on a project.

If you want to stick to reading about technologies and how they work and not implement them and see them actually work - maybe engineering isn’t for you. You can stick to academia instead. And there’s nothing wrong in that. Not everyone in the computer science community needs to be a great programmer. It’s just a subset.

Experience is a bigger teacher than training.

Dedipyaman

How Pycon India 2018 changed the way I see things

2018-10-29T00:00:00+00:00

I was all set up, I grabbed my backpack, put my laptop in and ready to go. We reached the railway station by 4 PM and had the train at 5:30 to Hyderabad. I was excited. Firstly because I was taking a few days off college after 3 months, secondly I am going to visit this Pycon thing which I heard about from my friend Kuntal a few months back.

I originally planned to get the student tickets, but got caught up with the schedule and had to get the regular ones. It’s alright, I thought, I just hope the experience (or the food) makes up for it. We reached Hyderabad by 12 midnight, and stayed at my roommate’s place. Crashed on the bed by 1. I was excited and couldn’t sleep for a while. But my body gave in to the fatigue of the travel.

Day 1

I dressed to my best in the next morning. We took a cab, I was enjoying the Hyderabadi streets and traffic. As we approached HICC, I saw the Pycon posters - enter nervousness. Amongst all the excitement to meet developers with similar interests, I realized:

I am not a regular Python developer.
This is my first meet up and I haven’t done anything like this before.

I wasn’t very sure how I’d connect to people when the one thing that brought them together - Python - wasn’t my go-to language for anything. I barely ever used python. Even when I had to work on a project with a Raspberry Pi, I wrote the entire application in Java and ran a few tiny python scripts on top of it, but that’s it.

Upon reaching the beautiful HICC, I saw people, all sorts of people walking around with their Pycon IDs and clicking photos, talking to each other. Nothing unusual. You see that everyday.

But that was not a regular crowd.

That was a passionate group of people who loved doing what they did. They were developers of all ages, all races. Some of them came from far off places. They had been united by one common interest - writing software, making a difference and sharing ideas.

I was thrilled and scared at the same time. What if I am not good enough? What if they don’t accept me as one of them? But damn it, I paid for this and I travelled far enough.

I reached the counter, looking around at people with clothes that proudly said that they were developers. A Hacktoberfest T-shirt here, a Linux t-shirt there. I got my ID, met Kuntal and grabbed my goodies and stickers(!).

Armin Ronacher at Pycon 2018

I attended the first Keynote speech by Armin Ronacher, the creator of Flask! That is celebrity level. He talked about how the Python and the Rust community evolved and what the Python community can learn from the Rust community.

After the amazing speech, I turned to the audience and I saw hundreds and hundreds of developers. Young and old. It was overwhelming.

I started visiting the stalls. This was my chance. Connect, I said to myself. I found out about the job opportunities. I met some really cool people in the process. I visited the open source communities and worked some programming contests and quizzes.

I realized I was wrong. The community made me feel welcome at home. It felt like I am a part of it. No more an alien. I was one of them.

I went to the Pyladies and the LinuxChix India community stalls and that was the first time I met so many lady programmers who were genuinely passionate about it. They have been doing a great job encouraging more women into technology.

Then, there was the FOSSASIA stall. I met Mario Behling and his team demonstrating the PSLAB there. They also organized Jugaad fest, Codeheat and a lot more! I couldn’t believe that I was talking to the founder of FOSSASIA. They were making wonderful stuff, open for all and yet so down to earth and genuine.

In the PSF stall, I was curious about this GitCoin thing on the stickers, and out of nowhere a guy with glasses - who I later found out was Kushal Das- stepped in and said, “Gitcoin is about paying developers for contributing to open source. Open source is all about money. If anyone says otherwise, DON’T BELIEVE THEM!” I wasn’t so sure how to react for a moment or two. This was interesting!

I kept hopping stalls for a while and came lunchtime.

Pycon India 2018 had one of the best buffets I have ever had. All the variety, all the freshly made stuff, all for you. It couldn’t have been better.

I attended a few more speeches, learnt a lot about optimzing applications, architecting solutions efficiently and how software can create a huge impact on the society- with the power of open source and collaboration. I got the chance to listen to the lightning talks after that.

I was amazed to see what people were doing with software!

Day 2

I came back excited this time, attended the keynote by Travis Oliphant.

I couldn’t believe who stood before my eyes. Although I am not a regular Pythonista. I have heard about and seen Numpy, Scipy and Anaconda in action. The guy who created them all stood before me.

Travis Oliphant's keynote speech at Pycon India 2018

He shared some wonderful insights on how things changed over the years, how the community grew and how he evolved with it. He shared his journey from being a scientist to be the founder of Quansight. And the importance of family and relationships.

I later had the chance to meet him personally. He gave me some valuable insights on how to manage time effectively and how to think while building something. How to look for certain things and going into the details.

Travis Oliphant and Dedipyaman Das

So, I have met someone who literally changed the lives of millions. Something to be taken off my bucket list.

I later got the chance to meet Armin Ronacher as well who shared some great tips about what and what not to worry about while building web applications.

Although I wasn’t a Pythonista, it all made sense to me!

Hopped a few more stalls, met few people at the ILUGD. Everyone around me were friendly creatures. Open minded. This was all overwhelming.

The conference of Pycon India was concluded with Carol Willing, core developer of Jupyter and CPython, and her inspiring speech about the future of Python. It was wonderful. She took us back in time through the entire conference once again. Recaped the moments we had in the conference.

After attending the lightning talks, my feet felt heavy. I didn’t want to go back. After all of this, this is where I wanted to be. I had two more days left.

Day 3 and 4 : The sprints

The developer sprints helped me connect, work with the community and get to know how open source works better. Although, I have been an open source contributor for sometime, the dev sprints really gave me a much needed push.

I have been slacking without motivation for a month before that. Not having written any code for a month, I wasn’t very confident. I just wanted to make sure that I didn’t embarass myself.

I joined the Jupyter team with Carol Willing and Miss Madhu mentoring the developers for iPywidgets and other Jupyter applications. I decided to work on jupyter/qtconsole, since I had worked on some Qt applications, albeit they were on C++.

I started reading some code and documentation. I was surprised how fast I could understand whatever was happening inside. Python is a really easy and welcoming language I figured.

I went up to Carol, sat next to her and she gave me some nice pointers to get started off with it. I sat with a developer with 41 years of experience and few other developers and coded.

Wait, this is what I have been aspiring to do for the last 8 years of my programming career!

Developing with Jupyter and Carol

This was dream coming true.

I got my first pull request accepted and merged by Carol to qtconsole. And the feeling was wonderful.

This, this changed everything. I finally got a reason to start programming once again. I started looking out for more probable bug fixes, documentation changes that could help the projects.

I even got the chance to talk to her on a human level. She shared her story - how things began from her home near IBM when she started coding in her fifth grade. From Basic to Processing to Python. The long way she has come since then.

We had lunch together, on the floor.

And we clicked selfies!

Dedipyaman's selfie with Carol Willing

I had so much to learn from this woman! This is how my heroes look like.

After two days of dev-sprints (which also helped me grab a chance for Hacktoberfest t-shirts), I had to return to my college.

Although, I did want more. I was content. I was happy. This has changed me. This is what open source is all about. It’s about creating connections, which leads to wonderful results.

These days I have been writing code and contributing to open source almost everyday.

I have met people who have impacted the lives of millions. They talked to people for hours. Sharing ideas, thoughts, experiences. They weren’t bothered. They seemed happy to share it all. I don’t know how they do it, but they did.

I would come back here hundred more times if I could. But I guess I won’t be alive for that long. Alas.

I couldn’t have asked for more. Thank you Pycon. For the t-shirt, the goodies, the food and the experience.

If you are a programmer who hasn’t visited Pycon yet, do yourself a favor, visit Pycon. The experience would be worth remembering and sharing. I will see you there next year!

I realized, it was never just about Python. It has always been about the community.

Dedipyaman

The engineering addiction

2018-09-09T00:00:00+00:00

When I decided to leave Facebook and Instagram for good, more than a year back, I had a valid point- social media was getting toxic. Unbearable. It was growing heavy on me.

I was having trouble dealing with my priorities, living in a world of lies, where every other person was happy. Their smiles on social media would brighten up anyone’s day.

They had the perfect life. I was confused.

How can I be not so happy? Why doesn’t my profile reflect the happiness that they do?

Well, then I decided to check my own profile, and guess what?

I was happy too! At least my profile made it look like I was.

Social media is not what you are but what you make of it.

I was going through a range of emotions, not all at once. It wasn’t a bombshell dropped on me. It was more like poetry. Taking me through different facets, gradually progressing while slowly pouring on me with one central idea, which wasn’t so easy to decipher- just like poems are.

At one point when, I figured where I break. I found my tripping point.

I had mixed emotions. I had this idea, “what would happen if the fancy online presence I made for myself is not there when I would need it the most? Because someday I will come up with this world-changing thing that would need all the exposure I can give it to fly”.

Naturally, the someday never came. It never does.

With the gradual beating from life’s poetry, I gave up the thought of the one magical idea of the future, the what-if. I stopped using social media. I deleted Facebook, Instagram.

I went complete incognito.

I pulled myself out of the social media circle. I have grown a lot since then. I worked on a dozen new projects that helped me skill up, I came out of my comfort zone and wrote some applications, libraries that helped me grow in ways I couldn’t have imagine a year back. I have gained some valuable experience and I have learnt to communicate my ideas.

I didn’t need to care about whose birthday it was, or who just bought a fancy new iPhone. No more needing validation from pointless internet points in the forms of likes from people I barely cared about, but I pretended I did.

I eventually joined some of the essential social media platforms which I do need for communicating with people for work. I figured LinkedIn was a great medium to make great connections on a professional level, Twitter became a nice place to get updates from my favorite things to follow and I started checking my email instead of Whatsapp messages.

With all of these happening, I went full productive-workaholic mode. I needed to make something everyday. Make some significant change. Nothing in the world gave me more pleasure than making that one bit of code work. It grew on me. I wanted to acheive something everyday.

I slowly got addicted to productivity. It was my drug. Then came real life, and disappointments. The thing is, addiction is bad. Period. Even if it’s something good you are getting addicted to, if you don’t have enough flexibility in your mindset, you tend to disappoint yourself. Every single day, when I wouldn’t get something significant done - I would be sad. I would go to bed not happy with myself, introspecting everything wrong that I did today and hoping to have a better tomorrow.

This thing never worked. I have this problem, and I am sure a lot of others do too, if I am not productive throughout the entire day - I would have this literal heat passing from my feet to my body and I can feel it very well. It happens at night, usually after my dinner. My feet would turn cold, my palms would be sweaty and I would have this sadistic anger within me, that I almost want to break everything I have. And then, my friend who lives inside me, wakes up. A little demonic angel. This guy, comes up with a brilliant idea!

Look, you wasted the entire day, you didn’t acheive what you wanted to, and what you could have. You have 3 more hours of uptime left, and you will be weak and angry for this period. Why don’t you just sleep now, and I guarantee you, tomorrow is going to be the day when you change the world?

The guy sounds pretty damn reasonable. After all, what good can I do in the last 3 hours of the day when I couldn’t do anything at all the entire day? This, when coupled with procrastination would be lethal. The panic monkey visits me occasionally, I get productive in the last moments sometimes. Sometimes I don’t. This, when continued for a week or more - brings in a new thought process. A scary one.

I would get more sadistic and more destructive. Every ounce of motivation would be crippled, and I would crave the social life I once had. Motivation is the hardest to find at that point.

Focus and attention are two of the hardest earned things in my life. At the end, obsession towards productivity turned on me. The mind needs to be flexible enough to handle all the kind of curves life would throw at you. Perhaps the best way is to hustle, take every day as a different one. Things change, situations change. If you need to find a motivation, do new things and find that trigger.

Letting it grow, perhaps, isn’t the best option. Growing how to deal with it maybe is. It works for me at least. When I changed my mindset- to live one day at a time. I stopped looking for how I wasn’t productive so long, but what can I do in the next one hour? Maybe one tiny refactoring could help.

It turned out, I wasn’t as disappointed on myself as I would have been. I could do more. The motivation was to do one tiny thing instead of making a big change everyday. This worked out for me, pretty well.

Although I still have downtimes, I have low days. I know how to deal with them now. It’s easier when you do have a bigger goal in mind, but you take one small atomic step at a time.

Go easy on yourself. Life doesn’t need to be so strict. We don’t have to be a railway track when life itself is more like a curvy rocky mountain way.

Make atomic changes, atomic commits.

Dedipyaman

Learn programming in 24 hours!

2018-08-06T00:00:00+00:00

Every now and then I come across Youtube videos or ads along the lines of - “Learn programming in 24 hours!” or “Learn React.js in a week”. What’s the rush?

Programming is a craft. And like any form of craft, you don’t master it in 24 hours. You don’t master it in 24 days. Heck, you might not even master it in 24 years.

So, how do you really get there? Or do you even get there?

The magic sauce to this recipe is- *drumroll* practice. Ah, the glaringly obvious word.. What a disappointment, right? But really, think about it. How did the best programmer, muscian, driver, carpenter or anyone who you’d consider an expert get there?

Was it because they found some wonderful resource online or a youtube tutorial that helped them attain Nirvana? Did they learn it in 24 or 36 hours?

They started off just like any other kid who wants to ride a bicycle for the first time. By doing it. You didn’t read a manual on how to ride a bicycle in 24 hours when you were young, did you?

As a kid does, they tried out something that intrigued them. And like in most cases, they fell. They failed. Failure is an important step towards success. If you don’t know the taste of failure, you’d never know the joy of winning. And chances are, you will never win.

I understand, we are humans. We are greedy and we don’t like to risk it. Most of the undergrad students in my college took CS simply because “it has a good scope” or “it’s trending in the market now” or even for outrageous reasons like “I got great marks in my XYZ exam which involved math, physics and chemistry- so I should take CS because that’s what toppers do”. I am not saying that these are wrong motivations for joining CS, you gotta feed yourself, right? But when you do join CS, and then you DO NOT like the subject and you start memorizing code snippets, googling “learn C++ in 24 hours” to pass the test tomorrow, that’s where you are doing it wrong.

YOU DO NOT LEARN PROGRAMMING IN 24 HOURS. Repeat after me- YOU DO NOT LEARN PROGRAMMING IN 24 HOURS.

Sure, you can learn a few keywords, variable assignments, conditional statements, loops in a couple of hours. But that’s not remotely close to what programming really stands for. You are memorizing a language not learning how to program.

To be a programmer, to learn programming you have to start from the very beginning, go into the child curious mode and start asking the silliest questions that come to your mind. And then you start programming, with even the tiniest bit of information you gathered uptil now. You apply everything you learnt. You have to see it work and understand why it works. And then you learn a bit more, and you write a bit more. And you keep doing it. For a day. A week. A month. A year. A decade. A lifetime.

“Now take your best rhyme, outdo it, now do it a thousand times” - Eminem

You know the difference between a real software engineer at google and a wannabe hacker boyz straight out of college? The software engineer wrote thousands, maybe millions of lines of code in his or her life while our four year degree hackerz has been on youtube for far too long and installed Kali Linux. The skillful engineer is humble and knows their limitations. They keep learning and they don’t stop. The more they learn about something, the more they know how much they don’t know.

The only way to learn programming is by programming. That’s a recursive process and this is how every other form of craft or engineering is meant to be. You don’t learn if you don’t build enough.

“The master has failed more times than the beginner has even tried.” - Stephen McCranie

Over the years, I have been through a lot of phases on this journey to be a software developer. I have had the opportunity to meet some brilliant people and some downright desperate “coders”. The biggest differentiating factor between these two were the amount of time and effort each group put on themselves before they started calling them “programmers”. You need to write a lot of programs to become a good programmer.

So, you want to become a web/android/desktop app/xyz developer?

Here’s what you gotta do. Go out there, find a cool looking website, app or whatever, check out why it looks as such and draw a picture in your mind. This is your project for the next summer. Oh wait. Nope. “for the next summer” - very very bad suggestion. You DO NOT wait for the next summer. You do it now. Like, right now. You find a resource online for free, you follow it and you build something.

Don’t worry about good or bad practices. Don’t worry about how the end product looks. Your first piece of big project is going to be crappy. So is your next project. So is the next one. But that’s going to be relatively less crappier than every former version. It’s pretty common in the software business to write pretty code now and curse ourselves six months later for writing such a huge unmaintainable piece of crap. And that’s okay. You are most likely to fail on the very first project, and it’s okay.

Programming is not something you do in a day or a week. You take your time, commit to a task and keep improving everyday.

So how do I really get started?

My next blog post will contain some tips and pointers. Just a few hints to get you started. It won’t explain why this framework is better than the other or which language is the best, but it will give you a slight heads up on what to expect and what to avoid. I learnt from my mistakes, you probably don’t want to make the same mistakes that are easily avoidable and just downright silly. I will let you make your own sets of mistakes. Tune in to my next post to learn programming in 10 years.

Here is a link to the continuation of this post where I write about how I approached programming: Learn Programming the Right Way.

PS.: I prefer refering to what I do as “Engineering”. Programming is just one part of what I do, one piece of the puzzle.

Additional reads:

Teach yourself programming in 10 years - Peter Norvig

Dedipyaman

Twodee’s kitchen

Contextual exceptions

Design patterns - to do or not to?

Another Pycon, more community

PHP sucks, can it suck less?

Start writing OOP

Strict type as much as you can

Namespaces, please

Composer

Throw away mysql_*

Separate your concerns

PSRs to the rescue

A few concluding words

Technical debt

How do I prevent technical debt?

Don’t fail silently, scream!

Learn Programming The Right Way

Find your motivation

Stop reading! Start writing iteratively, learn from your experience

Add development tools to your arsenal and make life easier

Listen to your tools

Write programs, a lot of them!

How Pycon India 2018 changed the way I see things

Day 1

Day 2

Day 3 and 4 : The sprints

The engineering addiction

Learn programming in 24 hours!

So, you want to become a web/android/desktop app/xyz developer?

So how do I really get started?

Throw away `mysql_*`