Tech Guides - Programming

There is a rift in the tech landscape that has been shifting quietly for some time. But 2018 is the year it has finally properly opened. This is the rift between the tech’s entrepreneurial ‘superstars’ and a nascent solidarity movement, both of which demonstrate the two faces of the modern tech industry. But within this ‘culture war’ there’s a broader debate about what technology is for and who has the power to make decisions about it. And that can only be a good thing - this is a conversation we’ve needed for some time. With the Cambridge Analytica scandal, and the shock election results to which it was tied, much contemporary political conversation is centered on technology’s impact on the social sphere. But little attention has been paid to the way these social changes or crises are actually enforcing changes within the tech industry itself. If it feels like we’re all having to pick sides when it comes to politics, the same is true when it comes to tech. The rise of the tech ego If you go back to the early years of software, in the early part of the twentieth century, there was little place for ego. It’s no accident that during this time computing was feminized - it was widely viewed as administrative. It was only later that software became more male dominated, thanks to a sexist cultural drive to establish male power in the field. This was arguably the start of egos tech takeover- after all, men wanted their work to carry a certain status. Women had to be pushed out to give them it. It’s no accident that the biggest names in technology - Bill Gates, Steve Wozniak, Steve Jobs - are all men. Their rise was, in part, a consequence of a cultural shift in the sixties. But it’s recognise the fact that in the eighties, these were still largely faceless organizations. Yes, they were powerful men, but the organizations they led were really just the next step out from the military industrial complex that helped develop software as we know it today. It was only when ‘tech’ properly entered the consumer domain that ego took on a new value. As PCs became part of every day life, attaching these products to interesting and intelligent figures was a way of marketing these products. It’s worth remarking that it isn’t really important whether these men had huge egos at all. All that matters is that they were presented in that way, and granted an incredible amount of status and authority. This meant that complexity of software and the literal labor of engineering could be reduced to a relatable figure like Gates or Jobs. We can still feel the effects of that today: just think of the different ways Apple and Microsoft products are perceived. Tech leaders personify technology. They make it marketable. Perhaps tech ‘egos’ were weirdly necessary. Because technology was starting to enter into everyone’s lives, these figures - as much entrepreneurs as engineers - were able to make it accessible and relatable. If that sounds a little far fetched, consider what the tech ‘ninja’ or the ‘guru’ really means for modern businesses. It often isn’t so much about doing something specific, but instead about making the value and application of those technologies clear, simple, and understandable. When companies advertise for these roles using this sort of language they’re often trying to solve an organizational problem as much as a technical one. That’s not to say that being a DevOps guru at some middling eCommerce company is the same as being Bill Gates. But it is important to note how we started talking in this way. Similarly, not everyone who gets called a ‘guru’ is going to have a massive ego (some of my best friends are cloud gurus!), but this type of language does encourage a selfish and egotistical type of thinking. And as anyone who’s worked in a development team knows, that can be incredibly dangerous. From Zuckerberg to your sprint meeting - egos don’t care about you Today, we are in a position where the discourse of gurus and ninjas is getting dangerous. This is true on a number of levels. On the one hand we have a whole new wave of tech entrepreneurs. Zuckerberg, Musk, Kalanick, Chesky, these people are Gates and Jobs for a new generation. For all their innovative thinking, it’s not hard to discern a certain entitlement from all of these men. Just look at Zuckerberg and his role in the Cambridge Analytica Scandal. Look at Musk and his bizarre intervention in Thailand. Kalanick’s sexual harassment might be personal, but it reflects a selfish entitlement that has real professional consequences for his workforce. Okay, so that’s just one extreme - but these people become the images of how technology should work. They tell business leaders and politicians that tech is run by smart people who ostensibly should be trusted. This not only has an impact on our civic lives but also on our professional lives too. Ever wonder why your CEO decides to spend big money on a CTO? It’s because this is the model of modern tech. That then filters down to you and the projects you don’t have faith in. If you feel frustrated at work, think of how these ideas and ways of describing things cascade down to what you do every day. It might seem small, but it does exist. The emergence of tech worker solidarity While all that has been happening, we’ve also seen a positive political awakening across the tech industry. As the egos come to dictate the way we work, what we work on, and who feels the benefits, a large group of engineers are starting to realize that maybe this isn’t the way things should be. Disaffection in Silicon Valley This year in Silicon Valley, worker protests against Amazon, Microsoft and Google have all had an impact on the way their companies are run. We don’t necessarily hear about these people - but they’re there. They’re not willing to let their code be used in ways that don’t represent them. The Cambridge Analytica scandal was the first instance of a political crisis emerging in tech. It wasn’t widely reported, but some Facebook employees asked to move across to different departments like Instagram or WhatsApp. One product designer, Westin Lohne, posted on Twitter that he had left his position saying “morally, it was extremely difficult to continue working there as a product designer.” https://twitter.com/westinlohne/status/981731786337251328 But while the story at Facebook was largely disorganized disaffection, at Google there was real organization against Project Maven. 300 Google employees signed a petition against the company’s AI initiative with the Pentagon. In May, a number of employees resigned over the issue. One is reported as saying “over the last couple of months, I’ve been less and less impressed with Google’s response and the way our concerns are being listened to.” Read next: Google employees quit over company’s continued Artificial Intelligence ties with the Pentagon A similar protest happened at Amazon, with an internal letter to Jeff Bezos protesting the use of Rekognition - Amazon’s facial recognition technology - by law enforcement agencies, including ICE. “Along with much of the world we watched in horror recently as U.S. authorities tore children away from their parents,” the letter stated, according to Gizmodo. “In the face of this immoral U.S. policy, and the U.S.’s increasingly inhumane treatment of refugees and immigrants beyond this specific policy, we are deeply concerned that Amazon is implicated, providing infrastructure and services that enable ICE and DHS.” Microsoft saw a similar protest, sparked, in part, by the shocking images of families being separated at the U.S./Mexico border. Despite the company distancing itself over ICE’s activities, many employees were vocal in their opposition. “This is the sort of thing that would make me question staying,” said one employee, speaking to Gizmodo. A shift in attitudes as tensions emerge True, when taken individually, these instances of disaffection may not look like full-blown solidarity. But together, it amounts to a changing consciousness across Silicon Valley. Of course, it wouldn’t be wrong to say that a relationship between tech, the military, and government has always existed. But the reason things are different is precisely because these tensions have become more visible, attitudes more prominent in public discourse. It’s worth thinking about these attitudes and actions in the context of hyper-competitive Silicon Valley where ego is the norm, and talent and flair is everything. Signing petitions carries with it some risk - leaving a well-paid job you may have spent years working towards is no simple decision. It requires a decisive break with the somewhat egotistical strand that runs through tech to make these sorts of decisions. While it might seem strange, it also shouldn’t be that surprising. If working in software demands a high level of collaboration, then collaboration socially and politically is really just the logical development from our professional lives. All this talk about ‘ninjas’, ‘gurus’ and geniuses only creates more inequality within the tech job market - whether you’re in Silicon Valley, Stoke, or Barcelona, or Bangalore, this language actually hides the skills and knowledge that are actually most valuable in tech. Read next: Don’t call us ninjas or rockstars, say developers Where do we go next? The future doesn’t look good. But if the last six months or so are anything to go by there are a number of things we can do. On the one hand more organization could be the way forward. The publishing and media industries have been setting a great example of how unionization can work in a modern setting and help workers achieve protection and collaborative power at work. If the tech workforce is going to grow significantly over the next decade, we’re going to see more unionization. We’ve already seen technology lead to more unionization and worker organization in the context of the gig economy - Deliveroo and Uber drivers, for example. Gradually it’s going to return to tech itself. The tech industry is transforming the global economy. It’s not immune from the changes it’s causing. But we can also do more to challenge the ideology of the modern tech ego. Key to this is more confidence and technological literacy. If tech figureheads emerge to make technology marketable and accessible, the way to remove that power is to demystify it. It’s to make it clear that technology isn’t a gift, the genius invention of an unfathomable mind, but instead that it’s a collaborative and communal activity, and a skill that anyone can master given the right attitude and resources. At its best, tech culture has been teaching the world that for decades. Think about this the next time someone tells you that technology is magic. It’s not magic, it’s built by people like you. People who want to call it magic want you to think they’re a magician - and like any other magician, they’re probably trying to trick you.

Over the last couple of weeks the world has been watching on as rescuers attempted to find, and then save, a young football team from Tham Luang caves in Thailand. Owing to a remarkable coordinated effort, and a lot of bravery from the team (including one diver who died), all 12 boys were brought back to safety. Tech played a big part in the rescue mission too - from drones to subterranean radios. But it wanted to play a bigger role - or at least Elon Musk wanted it to. Musk and his submarine has been a somewhat bizarre subplot to this story, and while you can't fault someone for offering to help out in a crisis, you might even say it was unnecessary. Put simply, Elon Musk's involvement in this story is a fable about the worst aspects of tech-solutionism. It offers an important lesson for anyone working in tech how not to solve problems. Bringing a tiny submarine to a complex rescue mission that requires coordination between a number of different agencies, often operating from different countries is a bit like telling someone to use Angular to build their first eCommerce store. It's like building an operating system from scratch because your computer has crashed. Basically, you just don't need it. There are better and more appropriate solutions - like Shopify or WooCommerce, or maybe just rebooting your system. Lesson 1: Don't insert yourself in problems if you're not needed Elon Musk first offered his support to the rescue mission in Thailand on July 4. It was a response to one of his followers. https://twitter.com/elonmusk/status/1014509856777293825 Musk's first instincts were measures, saying that he suspects 'the Thai government has got this under control' but it didn't take long for his mind to change. Without any specific invitation or coordination with the parties leading the rescue mission, Musk's instincts to innovate and create kicked in. This sort of situation is probably familiar to anyone who works in tech - or, for that matter, anyone who has ever had a job. Perhaps you're the sort of person who hears about a problem and your immediate instinct is to fix it. Or perhaps you've been working on a project, someone hears about it, and immediately they're trying to solve all the problems you've been working on for weeks or months. Yes, sometimes it's appealing, but on the other side it can be incredibly annoying and disruptive. This is particularly true in software engineering where you're trying to solve problems at every level - from strategy to code. There's rarely a single solution. There's always going to be a difference of opinion. At some point we need to respect boundaries and allow the right people to get on with the job. Lesson 2: Listen to the people involved and think carefully about the problem you're trying to solve One of the biggest challenges in problem solving is properly understanding the problem. It's easy to think you've got a solution after a short conversation about a problem but there may be nuances you've missed or complexities that aren't immediately clear. Humility can be a very valuable quality when problem solving. It allows everyone involved to think clearly about the task at hand; it opens up space for better solutions. As the old adage goes, when every problem looks like a nail, every solution looks like a hammer. For Musk, when a problem looks like kids stuck in an underwater cave, the solution looks like a kid-sized submarine. Never mind that experts in Thailand explained that the submarine would not be 'practical.' For Musk, a solution is a solution. "Although his technology is good and sophisticated it’s not practical for this mission" said Narongsak Osatanakorn, one of the leaders of the rescue mission, speaking to the BBC and The Guardian. https://twitter.com/elonmusk/status/1016110809662066688 Okay, so perhaps that's a bit of a facetious example - but it is a problem we can run into, especially if we work in software. Sometimes you don't need to build a shiny new SPA - your multi-page site might be just fine for its purpose. And maybe you don't need to deploy on containers - good old virtual machines might do the job for you. In these sort of instances it's critical to think about the problem at hand. To do that well you also need to think about the wider context around it - what infrastructure is already there? If we change something, is that going to have a big impact on how it's maintained in the future? In many ways, the lesson here recalls the argument put forward by the Boring Software Manifesto in June. In it, the writer argued in favor of things that are 'simple and proven' over software that is 'hyped and volatile'. Lesson 3: Don't take it personally if people decline your solutions Problem solving is a collaborative effort, as we've seen. Offering up solutions is great - but it's not so great when you react badly to rejection. https://twitter.com/elonmusk/status/1016731812159254529 Hopefully, this doesn't happen too much in the workplace - but when your job is to provide solutions, it doesn't help anyone to bring your ego into it. In fact, it indicates selfish motives behind your creative thinking. This link between talent, status and ego has been developing for some time now in the tech world. Arguably Elon Musk is part of a trend of engineers - ninjas, gurus, wizards, whatever label you want to place on yourself - for whom problem-solving is as much an exercise in personal branding as it is actually about solving problems. This trend is damaging for everyone - it not only undermines people's ability to be creative, it transforms everyone's lives into a rat race for status and authority. That's not only sad, but also going to make it hard to solve real problems. Lesson 4: Sometimes collaboration can be more inspiring than Elon Musk Finally, let's think about the key takeaway here: everyone in that cave was saved. And this wasn't down to some miraculous invention. It was down to a combination of tools - some of them even pretty old. It wasn't down to one genius piece of engineering, but instead a combination of creative thinking and coordinated problem solving that used the resources available to bring a shocking story to a positive conclusion. Working in tech isn't always going to be a matter of life and death - but it's the collaborative and open world we want to work in, right?

A design pattern is a reusable solution to a recurring problem in software design. It is not a finished piece of code but a template that helps to solve a particular problem or family of problems. In this article, we will talk about the Gang of Four design patterns. The gang of four, authors Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides, initiated the concept of Design Pattern in Software development. These authors are collectively known as Gang of Four (GOF). We are going to focus on the design patterns from the Scala point of view. All different design patterns can be grouped into the following types: Creational Structural Behavioral These three groups contain the famous Gang of Four design patterns.  In the next few subsections, we will explain the main characteristics of the listed groups and briefly present the actual design patterns that fall under them. This article is an excerpt from Scala Design Patterns - Second Edition by Ivan Nikolov. In this book, you will learn how to write efficient, clean, and reusable code with Scala. Creational design patterns The creational design patterns deal with object creation mechanisms. Their purpose is to create objects in a way that is suitable to the current situation, which could lead to unnecessary complexity and the need for extra knowledge if they were not there. The main ideas behind the creational design patterns are as follows: Knowledge encapsulation about the concrete classes Hiding details about the actual creation and how objects are combined We will be focusing on the following creational design patterns in this article: The abstract factory design pattern The factory method design pattern The lazy initialization design pattern The singleton design pattern The object pool design pattern The builder design pattern The prototype design pattern The following few sections give a brief definition of what these patterns are. The abstract factory design pattern This is used to encapsulate a group of individual factories that have a common theme. When used, the developer creates a specific implementation of the abstract factory and uses its methods in the same way as in the factory design pattern to create objects. It can be thought of as another layer of abstraction that helps to instantiate classes. The factory method design pattern This design pattern deals with the creation of objects without explicitly specifying the actual class that the instance will have—it could be something that is decided at runtime based on many factors. Some of these factors can include operating systems, different data types, or input parameters. It gives developers the peace of mind of just calling a method rather than invoking a concrete constructor. The lazy initialization design pattern This design pattern is an approach to delay the creation of an object or the evaluation of a value until the first time it is needed. It is much more simplified in Scala than it is in an object-oriented language such as Java. The singleton design pattern This design pattern restricts the creation of a specific class to just one object. If more than one class in the application tries to use such an instance, then this same instance is returned for everyone. This is another design pattern that can be easily achieved with the use of basic Scala features. The object pool design pattern This design pattern uses a pool of objects that are already instantiated and ready for use. Whenever someone requires an object from the pool, it is returned, and after the user is finished with it, it puts it back into the pool manually or automatically. A common use for pools are database connections, which generally are expensive to create; hence, they are created once and then served to the application on request. The builder design pattern The builder design pattern is extremely useful for objects with many possible constructor parameters that would otherwise require developers to create many overrides for the different scenarios an object could be created in. This is different to the factory design pattern, which aims to enable polymorphism. Many of the modern libraries today employ this design pattern. As we will see later, Scala can achieve this pattern really easily. The prototype design pattern This design pattern allows object creation using a clone() method from an already created instance. It can be used in cases when a specific resource is expensive to create or when the abstract factory pattern is not desired. Structural design patterns Structural design patterns exist in order to help establish the relationships between different entities in order to form larger structures. They define how each component should be structured so that it has very flexible interconnecting modules that can work together in a larger system. The main features of structural design patterns include the following: The use of the composition to combine the implementations of multiple objects Help build a large system made of various components by maintaining a high level of flexibility In this article, we will focus on the following structural design patterns: The adapter design pattern The decorator design pattern The bridge design pattern The composite design pattern The facade design pattern The flyweight design pattern The proxy design pattern The next subsections will put some light on what these patterns are about. The adapter design pattern The adapter design pattern allows the interface of an existing class to be used from another interface. Imagine that there is a client who expects your class to expose a doWork() method. You might have the implementation ready in another class, but the method is called differently and is incompatible. It might require extra parameters too. This could also be a library that the developer doesn't have access to for modifications. This is where the adapter can help by wrapping the functionality and exposing the required methods. The adapter is useful for integrating the existing components. In Scala, the adapter design pattern can be easily achieved using implicit classes. The decorator design pattern Decorators are a flexible alternative to sub classing. They allow developers to extend the functionality of an object without affecting other instances of the same class. This is achieved by wrapping an object of the extended class into one that extends the same class and overrides the methods whose functionality is supposed to be changed. Decorators in Scala can be built much more easily using another design pattern called stackable traits. The bridge design pattern The purpose of the bridge design pattern is to decouple an abstraction from its implementation so that the two can vary independently. It is useful when the class and its functionality vary a lot. The bridge reminds us of the adapter pattern, but the difference is that the adapter pattern is used when something is already there and you cannot change it, while the bridge design pattern is used when things are being built. It helps us to avoid ending up with multiple concrete classes that will be exposed to the client. You will get a clearer understanding when we delve deeper into the topic, but for now, let's imagine that we want to have a FileReader class that supports multiple different platforms. The bridge will help us end up with FileReader, which will use a different implementation, depending on the platform. In Scala, we can use self-types in order to implement a bridge design pattern. The composite design pattern The composite is a partitioning design pattern that represents a group of objects that are to be treated as only one object. It allows developers to treat individual objects and compositions uniformly and to build complex hierarchies without complicating the source code. An example of composite could be a tree structure where a node can contain other nodes, and so on. The facade design pattern The purpose of the facade design pattern is to hide the complexity of a system and its implementation details by providing the client with a simpler interface to use. This also helps to make the code more readable and to reduce the dependencies of the outside code. It works as a wrapper around the system that is being simplified and, of course, it can be used in conjunction with some of the other design patterns mentioned previously. The flyweight design pattern The flyweight design pattern provides an object that is used to minimize memory usage by sharing it throughout the application. This object should contain as much data as possible. A common example given is a word processor, where each character's graphical representation is shared with the other same characters. The local information then is only the position of the character, which is stored internally. The proxy design pattern The proxy design pattern allows developers to provide an interface to other objects by wrapping them. They can also provide additional functionality, for example, security or thread-safety. Proxies can be used together with the flyweight pattern, where the references to shared objects are wrapped inside proxy objects. Behavioral design patterns Behavioral design patterns increase communication flexibility between objects based on the specific ways they interact with each other. Here, creational patterns mostly describe a moment in time during creation, structural patterns describe a more or less static structure, and behavioral patterns describe a process or flow. They simplify this flow and make it more understandable. The main features of behavioral design patterns are as follows: What is being described is a process or flow The flows are simplified and made understandable They accomplish tasks that would be difficult or impossible to achieve with objects In this article, we will focus our attention on the following behavioral design patterns: The value object design pattern The null object design pattern The strategy design pattern The command design pattern The chain of responsibility design pattern The interpreter design pattern The iterator design pattern The mediator design pattern The memento design pattern The observer design pattern The state design pattern The template method design pattern The visitor design pattern The following subsections will give brief definitions of the aforementioned behavioral design patterns. The value object design pattern Value objects are immutable and their equality is based not on their identity, but on their fields being equal. They can be used as data transfer objects, and they can represent dates, colors, money amounts, numbers, and more. Their immutability makes them really useful in multithreaded programming. The Scala programming language promotes immutability, and value objects are something that naturally occur there. The null object design pattern Null objects represent the absence of a value and they define a neutral behavior. This approach removes the need to check for null references and makes the code much more concise. Scala adds the concept of optional values, which can replace this pattern completely. The strategy design pattern The strategy design pattern allows algorithms to be selected at runtime. It defines a family of interchangeable encapsulated algorithms and exposes a common interface to the client. Which algorithm is chosen could depend on various factors that are determined while the application runs. In Scala, we can simply pass a function as a parameter to a method, and depending on the function, a different action will be performed. The command design pattern This design pattern represents an object that is used to store information about an action that needs to be triggered at a later time. The information includes the following: The method name The owner of the method Parameter values The client then decides which commands need to be executed and when by the invoker. This design pattern can easily be implemented in Scala using the by-name parameters feature of the language. The chain of responsibility design pattern The chain of responsibility is a design pattern where the sender of a request is decoupled from its receiver. This way, it makes it possible for multiple objects to handle the request and to keep logic nicely separated. The receivers form a chain where they pass the request and, if possible, they process it, and if not, they pass it to the next receiver. There are variations where a handler might dispatch the request to multiple other handlers at the same time. This somehow reminds us of function composition, which in Scala can be achieved using the stackable traits design pattern. The interpreter design pattern The interpreter design pattern is based on the ability to characterize a well-known domain with a language with a strict grammar. It defines classes for each grammar rule in order to interpret sentences in the given language. These classes are likely to represent hierarchies as grammar is usually hierarchical as well. Interpreters can be used in different parsers, for example, SQL or other languages. The iterator design pattern The iterator design pattern is when an iterator is used to traverse a container and access its elements. It helps to decouple containers from the algorithms performed on them. What an iterator should provide is sequential access to the elements of an aggregate object without exposing the internal representation of the iterated collection. The mediator design pattern This pattern encapsulates the communication between different classes in an application. Instead of interacting directly with each other, objects communicate through the mediator, which reduces the dependencies between them, lowers the coupling, and makes the overall application easier to read and maintain. The memento design pattern This pattern provides the ability to roll back an object to its previous state. It is implemented with three objects—originator, caretaker, and memento. The originator is the object with the internal state; the caretaker will modify the originator, and a memento is an object that contains the state that the originator returns. The originator knows how to handle a memento in order to restore its previous state. The observer design pattern This design pattern allows the creation of publish/subscribe systems. There is a special object called subject that automatically notifies all the observers when there are any changes in the state. This design pattern is popular in various GUI toolkits and generally where event handling is needed. It is also related to reactive programming, which is enabled by libraries such as Akka. We will see an example of this towards the end of this book. The state design pattern This design pattern is similar to the strategy design pattern, and it uses a state object to encapsulate different behavior for the same object. It improves the code's readability and maintainability by avoiding the use of large conditional statements. The template method design pattern This design pattern defines the skeleton of an algorithm in a method and then passes some of the actual steps to the subclasses. It allows developers to alter some of the steps of an algorithm without having to modify its structure. An example of this could be a method in an abstract class that calls other abstract methods, which will be defined in the children. The visitor design pattern The visitor design pattern represents an operation to be performed on the elements of an object structure. It allows developers to define a new operation without changing the original classes. Scala can minimize the verbosity of this pattern compared to the pure object-oriented way of implementing it by passing functions to methods. Choosing a design pattern As we already saw, there are a huge number of design patterns. In many cases, they are suitable to be used in combinations as well. Unfortunately, there is no definite answer regarding how to choose the concept of designing our code. There are many factors that could affect the final decision, and you should ask yourselves the following questions: Is this piece of code going to be fairly static or will it change in the future? Do we have to dynamically decide what algorithms to use? Is our code going to be used by others? Do we have an agreed interface? What libraries are we planning to use, if any? Are there any special performance requirements or limitations? This is by no means an exhaustive list of questions. There is a huge amount of factors that could dictate our decision in how we build our systems. It is, however, really important to have a clear specification, and if something seems missing, it should always be checked first. By now, we have a fair idea about what a design pattern is and how it can affect the way we write our code. We've iterated through the most famous Gang of Four design patterns out there, and we have outlined the main differences between them. To know more on how to incorporate functional patterns effectively in real-life applications, read our book Scala Design Patterns - Second Edition. Implementing 5 Common Design Patterns in JavaScript (ES8) An Introduction to Node.js Design Patterns

Design patterns have provided many ways to simplify the development of software applications. Now that organizations are beginning to tackle applications that leverage new sources and types of big data, design patterns for big data are needed. These big data design patterns aim to reduce complexity, boost the performance of integration and improve the results of working with new and larger forms of data. This article intends to introduce readers to the common big data design patterns based on various data layers such as data sources and ingestion layer, data storage layer and data access layer. This article is an excerpt from Architectural Patterns by Pethuru Raj, Anupama Raman, and Harihara Subramanian. In this book, you will learn the importance of architectural and design patterns in business-critical applications. Data sources and ingestion layer Enterprise big data systems face a variety of data sources with non-relevant information (noise) alongside relevant (signal) data. Noise ratio is very high compared to signals, and so filtering the noise from the pertinent information, handling high volumes, and the velocity of data is significant. This is the responsibility of the ingestion layer. The common challenges in the ingestion layers are as follows: Multiple data source load and prioritization Ingested data indexing and tagging Data validation and cleansing Data transformation and compression The preceding diagram depicts the building blocks of the ingestion layer and its various components. We need patterns to address the challenges of data sources to ingestion layer communication that takes care of performance, scalability, and availability requirements. In this section, we will discuss the following ingestion and streaming patterns and how they help to address the challenges in ingestion layers. We will also touch upon some common workload patterns as well, including: Multisource extractor Multidestination Protocol converter Just-in-time (JIT) transformation Real-time streaming pattern Multisource extractor An approach to ingesting multiple data types from multiple data sources efficiently is termed a Multisource extractor. Efficiency represents many factors, such as data velocity, data size, data frequency, and managing various data formats over an unreliable network, mixed network bandwidth, different technologies, and systems: The multisource extractor system ensures high availability and distribution. It also confirms that the vast volume of data gets segregated into multiple batches across different nodes. The single node implementation is still helpful for lower volumes from a handful of clients, and of course, for a significant amount of data from multiple clients processed in batches. Partitioning into small volumes in clusters produces excellent results. Data enrichers help to do initial data aggregation and data cleansing. Enrichers ensure file transfer reliability, validations, noise reduction, compression, and transformation from native formats to standard formats. Collection agent nodes represent intermediary cluster systems, which helps final data processing and data loading to the destination systems. The following are the benefits of the multisource extractor: Provides reasonable speed for storing and consuming the data Better data prioritization and processing Drives improved business decisions Decoupled and independent from data production to data consumption Data semantics and detection of changed data Scaleable and fault tolerance system The following are the impacts of the multisource extractor: Difficult or impossible to achieve near real-time data processing Need to maintain multiple copies in enrichers and collection agents, leading to data redundancy and mammoth data volume in each node High availability trade-off with high costs to manage system capacity growth Infrastructure and configuration complexity increases to maintain batch processing Multidestination pattern In multisourcing, we saw the raw data ingestion to HDFS, but in most common cases the enterprise needs to ingest raw data not only to new HDFS systems but also to their existing traditional data storage, such as Informatica or other analytics platforms. In such cases, the additional number of data streams leads to many challenges, such as storage overflow, data errors (also known as data regret), an increase in time to transfer and process data, and so on. The multidestination pattern is considered as a better approach to overcome all of the challenges mentioned previously. This pattern is very similar to multisourcing until it is ready to integrate with multiple destinations (refer to the following diagram). The router publishes the improved data and then broadcasts it to the subscriber destinations (already registered with a publishing agent on the router). Enrichers can act as publishers as well as subscribers: Deploying routers in the cluster environment is also recommended for high volumes and a large number of subscribers. The following are the benefits of the multidestination pattern: Highly scalable, flexible, fast, resilient to data failure, and cost-effective Organization can start to ingest data into multiple data stores, including its existing RDBMS as well as NoSQL data stores Allows you to use simple query language, such as Hive and Pig, along with traditional analytics Provides the ability to partition the data for flexible access and decentralized processing Possibility of decentralized computation in the data nodes Due to replication on HDFS nodes, there are no data regrets Self-reliant data nodes can add more nodes without any delay The following are the impacts of the multidestination pattern: Needs complex or additional infrastructure to manage distributed nodes Needs to manage distributed data in secured networks to ensure data security Needs enforcement, governance, and stringent practices to manage the integrity and consistency of data Protocol converter This is a mediatory approach to provide an abstraction for the incoming data of various systems. The protocol converter pattern provides an efficient way to ingest a variety of unstructured data from multiple data sources and different protocols. The message exchanger handles synchronous and asynchronous messages from various protocol and handlers as represented in the following diagram. It performs various mediator functions, such as file handling, web services message handling, stream handling, serialization, and so on: In the protocol converter pattern, the ingestion layer holds responsibilities such as identifying the various channels of incoming events, determining incoming data structures, providing mediated service for multiple protocols into suitable sinks, providing one standard way of representing incoming messages, providing handlers to manage various request types, and providing abstraction from the incoming protocol layers. Just-In-Time (JIT) transformation pattern The JIT transformation pattern is the best fit in situations where raw data needs to be preloaded in the data stores before the transformation and processing can happen. In this kind of business case, this pattern runs independent preprocessing batch jobs that clean, validate, corelate, and transform, and then store the transformed information into the same data store (HDFS/NoSQL); that is, it can coexist with the raw data: The preceding diagram depicts the datastore with raw data storage along with transformed datasets. Please note that the data enricher of the multi-data source pattern is absent in this pattern and more than one batch job can run in parallel to transform the data as required in the big data storage, such as HDFS, Mongo DB, and so on. Real-time streaming pattern Most modern businesses need continuous and real-time processing of unstructured data for their enterprise big data applications. Real-time streaming implementations need to have the following characteristics: Minimize latency by using large in-memory Event processors are atomic and independent of each other and so are easily scalable Provide API for parsing the real-time information Independent deployable script for any node and no centralized master node implementation The real-time streaming pattern suggests introducing an optimum number of event processing nodes to consume different input data from the various data sources and introducing listeners to process the generated events (from event processing nodes) in the event processing engine: Event processing engines (event processors) have a sizeable in-memory capacity, and the event processors get triggered by a specific event. The trigger or alert is responsible for publishing the results of the in-memory big data analytics to the enterprise business process engines and, in turn, get redirected to various publishing channels (mobile, CIO dashboards, and so on). Big data workload patterns Workload patterns help to address data workload challenges associated with different domains and business cases efficiently. The big data design pattern manifests itself in the solution construct, and so the workload challenges can be mapped with the right architectural constructs and thus service the workload. The following diagram depicts a snapshot of the most common workload patterns and their associated architectural constructs: Workload design patterns help to simplify and decompose the business use cases into workloads. Then those workloads can be methodically mapped to the various building blocks of the big data solution architecture. Data storage layer Data storage layer is responsible for acquiring all the data that are gathered from various data sources and it is also liable for converting (if needed) the collected data to a format that can be analyzed. The following sections discuss more on data storage layer patterns. ACID versus BASE versus CAP Traditional RDBMS follows atomicity, consistency, isolation, and durability (ACID) to provide reliability for any user of the database. However, searching high volumes of big data and retrieving data from those volumes consumes an enormous amount of time if the storage enforces ACID rules. So, big data follows basically available, soft state, eventually consistent (BASE), a phenomenon for undertaking any search in big data space. Database theory suggests that the NoSQL big database may predominantly satisfy two properties and relax standards on the third, and those properties are consistency, availability, and partition tolerance (CAP). With the ACID, BASE, and CAP paradigms, the big data storage design patterns have gained momentum and purpose. We will look at those patterns in some detail in this section. The patterns are: Façade pattern NoSQL pattern Polyglot pattern Façade pattern This pattern provides a way to use existing or traditional existing data warehouses along with big data storage (such as Hadoop). It can act as a façade for the enterprise data warehouses and business intelligence tools. In the façade pattern, the data from the different data sources get aggregated into HDFS before any transformation, or even before loading to the traditional existing data warehouses: The façade pattern allows structured data storage even after being ingested to HDFS in the form of structured storage in an RDBMS, or in NoSQL databases, or in a memory cache. The façade pattern ensures reduced data size, as only the necessary data resides in the structured storage, as well as faster access from the storage. NoSQL pattern This pattern entails getting NoSQL alternatives in place of traditional RDBMS to facilitate the rapid access and querying of big data. The NoSQL database stores data in a columnar, non-relational style. It can store data on local disks as well as in HDFS, as it is HDFS aware. Thus, data can be distributed across data nodes and fetched very quickly. Let's look at four types of NoSQL databases in brief: Column-oriented DBMS: Simply called a columnar store or big table data store, it has a massive number of columns for each tuple. Each column has a column key. Column family qualifiers represent related columns so that the columns and the qualifiers are retrievable, as each column has a column key as well. These data stores are suitable for fast writes. Key-value pair database: A key-value database is a data store that, when presented with a simple string (key), returns an arbitrarily large data (value). The key is bound to the value until it gets a new value assigned into or from a database. The key-value data store does not need to have a query language. It provides a way to add and remove key-value pairs. A key-value store is a dictionary kind of data store, where it has a list of words and each word represents one or more definitions. Graph database: This is a representation of a system that contains a sequence of nodes and relationships that creates a graph when combined. A graph represents three data fields: nodes, relationships, and properties. Some types of graph store are referred to as triple stores because of their node-relationship-node structure. You may be familiar with applications that provide evaluations of similar or likely characteristics as part of the search (for example, a user bought this item also bought... is a good illustration of graph store implementations). Document database: We can represent a graph data store as a tree structure. Document trees have a single root element or sometimes even multiple root elements as well. Note that there is a sequence of branches, sub-branches, and values beneath the root element. Each branch can have an expression or relative path to determine the traversal path from the origin node (root) and to any given branch, sub-branch, or value. Each branch may have a value associated with that branch. Sometimes the existence of a branch of the tree has a specific meaning, and sometimes a branch must have a given value to be interpreted correctly. The following table summarizes some of the NoSQL use cases, providers, tools and scenarios that might need NoSQL pattern considerations. Most of this pattern implementation is already part of various vendor implementations, and they come as out-of-the-box implementations and as plug and play so that any enterprise can start leveraging the same quickly. NoSQL DB to Use Scenario Vendor / Application / Tools Columnar database Application that needs to fetch entire related columnar family based on a given string: for example, search engines SAP HANA / IBM DB2 BLU / ExtremeDB / EXASOL / IBM Informix / MS SQL Server / MonetDB Key Value Pair database Needle in haystack applications (refer to the Big data workload patterns given in this section) Redis / Oracle NoSQL DB / Linux DBM / Dynamo / Cassandra Graph database Recommendation engine: application that provides evaluation of Similar to / Like: for example, User that bought this item also bought ArangoDB / Cayley / DataStax / Neo4j / Oracle Spatial and Graph / Apache Orient DB / Teradata Aster Document database Applications that evaluate churn management of social media data or non-enterprise data Couch DB / Apache Elastic Search / Informix / Jackrabbit / Mongo DB / Apache SOLR Polyglot pattern Traditional (RDBMS) and multiple storage types (files, CMS, and so on) coexist with big data types (NoSQL/HDFS) to solve business problems. Most modern business cases need the coexistence of legacy databases. At the same time, they would need to adopt the latest big data techniques as well. Replacing the entire system is not viable and is also impractical. The polyglot pattern provides an efficient way to combine and use multiple types of storage mechanisms, such as Hadoop, and RDBMS. Big data appliances coexist in a storage solution: The preceding diagram represents the polyglot pattern way of storing data in different storage types, such as RDBMS, key-value stores, NoSQL database, CMS systems, and so on. Unlike the traditional way of storing all the information in one single data source, polyglot facilitates any data coming from all applications across multiple sources (RDBMS, CMS, Hadoop, and so on) into different storage mechanisms, such as in-memory, RDBMS, HDFS, CMS, and so on. Data access layer Data access in traditional databases involves JDBC connections and HTTP access for documents. However, in big data, the data access with conventional method does take too much time to fetch even with cache implementations, as the volume of the data is so high. So we need a mechanism to fetch the data efficiently and quickly, with a reduced development life cycle, lower maintenance cost, and so on. Data access patterns mainly focus on accessing big data resources of two primary types: End-to-end user-driven API (access through simple queries) Developer API (access provision through API methods) In this section, we will discuss the following data access patterns that held efficient data access, improved performance, reduced development life cycles, and low maintenance costs for broader data access: Connector pattern Lightweight stateless pattern Service locator pattern Near real-time pattern Stage transform pattern The preceding diagram represents the big data architecture layouts where the big data access patterns help data access. We discuss the whole of that mechanism in detail in the following sections. Connector pattern The developer API approach entails fast data transfer and data access services through APIs. It creates optimized data sets for efficient loading and analysis. Some of the big data appliances abstract data in NoSQL DBs even though the underlying data is in HDFS, or a custom implementation of a filesystem so that the data access is very efficient and fast. The connector pattern entails providing developer API and SQL like query language to access the data and so gain significantly reduced development time. As we saw in the earlier diagram, big data appliances come with connector pattern implementation. The big data appliance itself is a complete big data ecosystem and supports virtualization, redundancy, replication using protocols (RAID), and some appliances host NoSQL databases as well. The preceding diagram shows a sample connector implementation for Oracle big data appliances. The data connector can connect to Hadoop and the big data appliance as well. It is an example of a custom implementation that we described earlier to facilitate faster data access with less development time. Lightweight stateless pattern This pattern entails providing data access through web services, and so it is independent of platform or language implementations. The data is fetched through restful HTTP calls, making this pattern the most sought after in cloud deployments. WebHDFS and HttpFS are examples of lightweight stateless pattern implementation for HDFS HTTP access. It uses the HTTP REST protocol. The HDFS system exposes the REST API (web services) for consumers who analyze big data. This pattern reduces the cost of ownership (pay-as-you-go) for the enterprise, as the implementations can be part of an integration Platform as a Service (iPaaS): The preceding diagram depicts a sample implementation for HDFS storage that exposes HTTP access through the HTTP web interface. Near real-time pattern For any enterprise to implement real-time data access or near real-time data access, the key challenges to be addressed are: Rapid determination of data: Ensure rapid determination of data and make swift decisions (within a few seconds, not in minutes) before the data becomes meaningless Rapid analysis: Ability to analyze the data in real time and spot anomalies and relate them to business events, provide visualization, and generate alerts at the moment that the data arrived Some examples of systems that would need real-time data analysis are: Radar systems Customer services applications ATMs Social media platforms Intrusion detection systems Storm and in-memory applications such as Oracle Coherence, Hazelcast IMDG, SAP HANA, TIBCO, Software AG (Terracotta), VMware, and Pivotal GemFire XD are some of the in-memory computing vendor/technology platforms that can implement near real-time data access pattern applications: As shown in the preceding diagram, with multi-cache implementation at the ingestion phase, and with filtered, sorted data in multiple storage destinations (here one of the destinations is a cache), one can achieve near real-time access. The cache can be of a NoSQL database, or it can be any in-memory implementations tool, as mentioned earlier. The preceding diagram depicts a typical implementation of a log search with SOLR as a search engine. Stage transform pattern In the big data world, a massive volume of data can get into the data store. However, all of the data is not required or meaningful in every business case. The stage transform pattern provides a mechanism for reducing the data scanned and fetches only relevant data. HDFS has raw data and business-specific data in a NoSQL database that can provide application-oriented structures and fetch only the relevant data in the required format: Combining the stage transform pattern and the NoSQL pattern is the recommended approach in cases where a reduced data scan is the primary requirement. The preceding diagram depicts one such case for a recommendation engine where we need a significant reduction in the amount of data scanned for an improved customer experience. The implementation of the virtualization of data from HDFS to a NoSQL database, integrated with a big data appliance, is a highly recommended mechanism for rapid or accelerated data fetch. We discussed big data design patterns by layers such as data sources and ingestion layer, data storage layer and data access layer. To know more about patterns associated with object-oriented, component-based, client-server, and cloud architectures, read our book Architectural Patterns. Why we need Design Patterns? Implementing 5 Common Design Patterns in JavaScript (ES8) An Introduction to Node.js Design Patterns

If you’re new to programming or have just a bit of experience, you’re probably thoroughly confused, wondering whether what you’ve been told all this while was bogus! If you’re an experience developer, you’re probably laughing (or scorning) at the title by now, wondering if I was high when I wrote the article. What I’m about to tell you is something that I’ve seen happen, and could be professionally beneficial to you. Although, I must warn you that it’s not what everyone is going to approve of, so read further but implement at your own risk. Okay, so I was saying, learn the framework, not the language. I’m going to explain why to take this approach, keeping two sets of audience in mind. The first, are total newbies, who’re probably working in some X field and now want to switch roles but have realised that with all the buzz of automation and the rise of tech, the new role demands a basic understanding of programming. The latter are developers who probably have varied levels of experience with programming, and now want to get into a new job, which requires them to have a particular skill. Later I will clearly list down the benefits of taking this approach. Let’s take audience #1 first. You’re a young Peter Parker just bitten by the programming bug You’re completely new to programming and haven’t the slightest clue about what it’s all about. You can spend close to a month trying to figure out a programming language like maybe Python or Java and then try to build something with it. Or you could jump straight into learning a framework and building something out of it. Now, in both cases we’re going to assume that you’re learning from a book, a video course or maybe a simple online tutorial. When you choose to learn the framework and build something, you’re going with the move fast and break things approach, which according to me, is the best way that anyone can learn something new. Once you have something built in front of you, you’re probably going to remember it much easier than when you’re just learning something theoretical first and then tried to apply it in practice at a later stage. How to do it? Start by understanding your goals first. Where do you want to go from where you are currently at. Now if your answer was that you wanted to get into Web Development, to build websites for a living, you have your first answer. What you need to do next is to understand what skills your “dream” company is actually looking for. You’ll understand that from the Job Description and a little help from someone technical. If the skill is web development, look for the accompanying tools/frameworks. Say for example, you found it was Node. Start learning Node! Pick up a book/video/tutorial that will help you build something as you learn. Spend at least a good week getting used to stuff. Have it reviewed by someone knowledgeable and watch carefully as the person works. You’ll be able to relate quite easily to what is going on, and will pick up some really useful tips and tricks quickly. Keep practicing another week, you’ll start getting good at it. Why will it work? Well, to be honest, several organisations work primarily with frameworks on a number of projects, mainly because frameworks simplify the building of large applications. Very rarely will you find the need to work with the vanilla language. By taking the Framework-first approach, you’re gaining the skill, i.e. web development, fast, rather than worry about using the medium or tool that will enable you to build it. You’re not spending too much time on learning the foundations, which you may never use in your development. Another example - Say you’ve been longing to learn how to build games, but don’t know how to program. Plus C++ is a treacherous language for a newbie to learn. Don’t worry at all! Just start learning how to work with Unreal Engine or any other game engine/framework. Use its in-built features, like Blueprints, which allows you to drag and drop things to build your game, and voila! You have your very own game! ;) You’re a Ninja wielding a katana in Star Wars Now you’re the experienced one, you probably have a couple of apps under your belt and are looking to learn a new skill, maybe because that’s what your dream company is looking for. Let’s say you’re a web developer, who now wants to move into mobile or enterprise application development. You’re familiar with JavaScript but don’t really want to take the time to learn a new language like C#. Don’t learn it, then. Just learn Xamarin or .NET Core! In this case, you’re already familiar with how programming works, but all that you don’t know is the syntax and working of the new language, C#. When you jump straight into .NET Core, you’ll be able to pick up the nitty gritties much faster than if you were to learn C# first and then start with .NET Core. No harm done if you were to take that path, but you’re just slowing down your learning by picking up the language first. Impossible is what? I know for a fact that by now, half of you are itching to tear me apart! I encourage you to vent your frustration in the comments section below! :) I honestly think it’s quite possible for someone to learn how to build an application without learning the programming language. You could learn how to drive an automatic car first and not know a damn thing about gears, but you’re still driving, right? You don’t always need to know the alphabet to be able to hold a conversation. At this point, I could cross the line by saying that this is true even in the the latest most cutting edge tech domain: machine learning. It might be possible even for buddying Data Scientists to start using Tensorflow straight away without learning Python, but I’ll hold my horses there. Benefits of learning a Framework directly There are 4 main benefits of this approach: You’re learning to become relevant quickly, which is very important, considering the level of competition that’s out there You’re meeting the industry requirements of knowing how to work with the framework to build large applications You’re unconsciously applying a fail-fast approach to your learning, by building an application from scratch Most importantly, you’re avoiding all the fluff - the aspects you may never use in languages or maybe the bad coding practices that you will avoid altogether As I conclude, it’s my responsibility to advise you that not learning a language entirely can be a huge drawback. For example, suppose your framework doesn’t address the problem you have at hand, you will have to work around the situation by working with the vanilla language. So when I say forget the language, I actually mean for the time being, when you’re trying to acquire the new skill fast. But to become a true expert, you must learn to master both the language and framework together. So go forth and learn something new today! Should software be more boring? The “Boring Software” manifesto thinks so These 2 software skills subscription services will save you time – and cash Minko Gechev: “Developers should learn all major front end frameworks to go to the next level”

I remember when I was in college a few years ago, this was a question everyone was asking. People who were studying Computer Science were always scared of this happening. Although it’s ironic because knowing the above, they were still brave enough to get into Computer Science in the first place! Okay, on a serious note, this is a highly debated topic and the IT industry is labeled to be notorious for employee burnout. The harsh reality Honestly speaking, I have developer friends who earn pretty good salary packages, even those working at a junior level. However, just two in five of them are actually satisfied with their jobs. They seem to be heading towards burnout quite quickly, too quickly in fact. I would understand if you told me that a middle aged person, having certain health conditions et al, working in a tech company, was nearing burnout. Here I see people in their early 20’s struggling to keep up, wishing for the weekend to come! Facts and figures Last month, a workspace app called Blind surveyed over 11K (11,487 to be precise) employees in the tech industry and the responses weren’t surprising! At least for me. The question posed to them was pretty simple: Are you currently suffering from job burnout? Source: TeamBlind Oh yeah, that’s a whopping 6,566 employees! Here’s some more shocking stats: When narrowed down to 30 companies, 25 of them had an employee burnout rate of 50% or higher. Only 5 companies had an employee burnout rate below 50%. Moreover, 16 out of the 30 companies had an employee burnout rate that was higher than the survey average of 57.16%. While Netflix had the least number of employees facing burnout, companies like Credit Karma, Twitch and Nvidia recorded the highest. I thought I’d analyse a bit and understand what some of the most common reasons causing burnout in the tech industry, could be. So here they are: #1 Unreasonable workload Now I know this is true for a fact! I’ve been working closely with developers and architects for close to 5 years now and I’m aware of how unreasonable projects can get. Especially their timelines. Customer expectation is something really hard to meet in the IT sector, mainly because the customer usually doesn’t know much about tech. Still, deadlines are set extremely tight, like a noose around developers’ necks, not giving them any space to maneuver whatsoever. Naturally, this will come down hard on them and they will surely experience burnout at some time, if not already. #2 Unreasonable managers In our recent Skill-Up survey, more than 60% of the respondents felt they knew more about tech, than what their managers did. More than 40% claimed that the biggest organisational barriers to their organisation’s (theirs as well) goals was their manager’s lack of tech knowledge. As with almost everyone, developers expect managers to be like a mentor, able to guide them into taking the right decisions and making the right choices. Rather, with the lack of knowledge, managers are unable to relate to their team members, ultimately coming across as unreasonable to them. On the other side of town, IT Management has been rated as one of the top 20 most stressful jobs in the world, by careeraddict! #3 Rapidly changing tech The tech landscape is one that changes ever so fast, and developers tend to get caught up in the hustle to stay relevant. I honestly feel the quote, “Time and tide wait for none” needs to be appended to “Time, tide and tech wait for none”! The competition is so high that if they don’t keep up, they’re probably history in a couple of years or so. I remember in the beginning of 2016, there was a huge hype about Data Science and AI - there was a predicted shortage of a million data scientists by 2018. Thousands of engineers all around the world started diving into their pockets to fund their Data Science Masters Degrees. All this can put a serious strain on their health and they ultimately meet burnout. #4 Disproportionate compensation Tonnes of software developers feel they’re underpaid, obviously leading them to lose interest in their work. Ever wonder why developers jump companies so many times in their careers? Now this stagnation is happening while on the other hand, work responsibilities are rising. There’s a huge imbalance that’s throwing employees off track. Chris Bolte, CEO of Paysa, says that companies recruit employees at competitive rates. But once they're on board, the companies don't tend to pay much more than the standard yearly increase. This is obviously a bummer and a huge demotivation for the employees. #5 Organisation culture The culture prevailing in tech organisations has a lot to do with how fast employees reach burnout. No employee wants to feel they’re tools or perhaps cogs in a wheel. They want to feel a sense of empowerment, that they’re making an impact and they have a say in the decisions that drive results. Without a culture of continuous learning and opportunities for professional and personal growth, employees are likely to be driven to burnout pretty quickly, either causing them to leave the organisation or worse still, lose confidence in themselves. #6 Work life imbalance This is a very tricky thing, especially if you’re out working long hours and you’re mostly unhappy at work. Moreover, developers usually tend to take their work home so that they can complete projects on time, and that messes up everything. When there’s no proper work life balance, you’re most probably going to run into a health problem, which will lead you to burnout, eventually. #7 Peer pressure This happens a lot, not just in the IT industry, but it’s more common here owing to the immense competition and the fast pace of the industry itself. Developers will obviously want to put in more efforts than they can, simply because their team members are doing it already. This can go two ways: One where their efforts still go unnoticed, and secondly, although they’re noticed, they’ve lost on their health and other important aspects of life. By the time they think of actually doing something innovative and productive, they’ve crashed and burned. [dropcap]I[/dropcap]f you ask me, burnout is a part and parcel of every industry and it majorly depends on mindset. The mindset of employees as well as the employer. Developers should try avoiding long work hours as far as possible, while trying to take their minds off work by picking up a nice hobby and exploring more ways to enrich their lives. On the other side of the equation, employers and managers should do better at understanding their team’s limitations or problems, while also maintaining an unbiased approach towards the whole team. They should realize that a motivated and balanced team is great for their balance sheet in the long run. They must be serious enough to include employee morale and nurturing a great working environment as one of management’s key performance indicators. If the IT industry must rise as a phoenix from the ashes, it will take more than a handful of people or organizations changing their ways. Change begins from within every individual and at the top for every organization. Should software be more boring? The “Boring Software” manifesto thinks so These 2 software skills subscription services will save you time – and cash Don’t call us ninjas or rockstars, say developers  

The year is 2018 and it’s all over the television, the internet, the newspapers; people are talking about it in coffee shops, at office desks across from where we sit, and what not. There’s a scramble for people to learn how to program. It’s a confusing and scary situation for someone who has never written a line of code, to think about all these discussions that are doing the rounds. In this article, I’m going to give you 5 reasons why I think you should learn to code, even if you are not a programmer by profession. Okay, first thing’s first: What is Programming? Programming is the process of writing/creating a set of instructions that tell a computer how to perform a certain task. Just like you would tell someone to do something and you would tell them in a language like English, computers also understand particular languages. This is called a programming language. There are several like Java, Python, C# (pronounced Csharp), etc. Just like many would find English easier to learn that French or maybe Cantonese, every person finds each language different, although almost all languages can do pretty much the same thing. So now, let’s see what our top 5 reasons are to learn a programming language, and ultimately, how to program a computer. #1 Automate stuff: How many times do we find ourselves doing the same old monotonous work ourselves. For example, a salesperson who has a list of 100 odd leads, will normally mail each person manually. How cool would it be if you could automate that and let your computer send each person a mail separately addressing them appropriately? Or maybe, you’re a manager who has a load of data you can’t really make sense of. You can use a language like Python to sort it and visualise your findings. Yes, that’s possible with programming! There’s a lot of other stuff that can be automated too, like HR scanning resumes manually. You can program your computer to do it for you, while you spend that time doing something more productive! Now while there might be softwares readily available that could do this for you, they’re pretty much standard and non-customisable. With programming, you can build something that’s tailor-made to your exact requirement. #2 Start thinking more logically: When you learn to program, you start thinking about outcomes more logically. Programming languages are all about logic and problem-solving. You will soon learn how to break down problems into small parts and tackle them individually. You can apply this learning in your own personal and work life. #3 Earn great moolah Programming pays really well and even freelance jobs pay close to $100 an hour. You could have your day job, while taking advantage of your programming skills to build websites, games, create applications for clients, after work or over the weekend, while making some good bucks. Here’s a list of average salaries earned by programmers, based on the language they used: Source: TOP 10 ChallengeRocket.com ranking of projected earnings in 2017 #4 Another great idea! Well, in case you’re an entrepreneur or are planning to become one, learning a programming language is sure to benefit you a great deal. The most successful startups these days are AI and software based and even though you might not be the one doing the programming, you will be interacting with those who will. It makes things much easier when you’re discussing with such a person, and more importantly, it saves you from being taken for a ride in many ways. #5 Having fun Unlike several other things that are boring to learn and will get you frustrated in a matter of hours, programming isn’t like that. That’s not to say that programming doesn’t have a learning curve, but with the right sources, you can learn it quickly and effectively. There are few things that can compare to the satisfaction of creating something. You can use programming to build your own game or maybe prank somebody! I tried that once - every time a friend clicked on the browser icon on my PC, it would make a loud farting noise! Don’t believe me yet? Over 80% of respondents to our most recent Skill-Up survey said that they programmed for fun, outside of work. #bonusreason! What’s to lose? I mean, seriously what can you lose? You’re going to be learning something completely new and will be probably much better at solving problems at home or your workplace. If you’re thinking you won’t find time to learn, think again. I’m sure all of us can make time, at least an hour a day to do something productive, if we commit to it. And you can always consider this your “me time”. Okay, so now you have your 5+1 reasons to learn to program. You’ve had some quality time to think about it and you’re ready to start learning. But you have some questions like where to start? Do you need to take a course or a college degree? Will it cost much? How long will it take to learn programming? The list is never ending. I’m going to put up some FAQs that most people ask me before they intend to start learning how to code. So here it goes… FAQs Where to start? Honestly speaking, you can start in the confines of your home! You just need a computer, an internet connection and the will to learn, if you want to get started with programming. You can begin by understanding what programming is a bit more, selecting a programming language, and then diving right into coding with the help of some material like the book, Introduction to Programming. What language do I pick? Every language can pretty much do what others can, but there are certain languages that have been built to solve a particular problem. Like for example, JavaScript, HTML and CSS are mainly used for building websites. Python is quite simple to learn and can be used to do a variety of things, most notably working with data. On the other hand, C# can be used to develop some cool games, while also being a great language to build websites and other applications. Think about what you want to do and then choose a language accordingly. I would suggest you choose between Python and JavaScript to start off. Do you need to take a course or a college degree? Not really, unless you plan on making it your full time career or becoming a software engineer or something like that. I’ve known some of the top professionals who haven’t earned a degree and still are at the position where they are. Mark Zuckerberg for example, dropped out of Harvard to start Facebook (he recently received an honorary degree in 2017, though). Programming is about learning to solve problems and in most cases, you don’t need a degree to prove that you’re great at solving problems. You can take an online course or buy a book to start learning. Sometimes, just looking at code often can teach you a lot too. Take HTML and CSS for example. If you like how a website looks, you could just checkout its source code to understand why it is the way it. Do this for a few sites and you you grasp the basics of what the HTML/CSS code do and how to write or alter simple code snippets. Will it cost much? You can learn a lot freely if you have a lot of time and patience at hand; sorting out the good from the bad. There are plenty of resources out there from Q&A sites like stackoverflow to youtube with its vast collection of videos. If you are like most people with a day job, you are better off spending a little to learn. There are several reasonably priced videos and courses from Packt, that will help you get started with computer programming. Alternatively, you can purchase a book or two for under $100. Trust me, once you become good at programming, you’ll be earning way more than you invested! How long will it take to learn programming? I can’t really answer that for certain. I took about 4 months to learn Python, while a friend of mine could code small programs within a couple of weeks. It all depends on the language you choose to learn, the amount of time you invest and how committed you are to learning something new. What jobs can I get? You may be quite happy in your current job as a non-programmer who now knows to code. But in case, you’re wondering about job prospects in programming, here is the rundown. As a programmer, you have a variety of jobs to choose from, depending on your area of interest. You could be a web developer, or a game developer, or you could also be building desktop applications like a notepad or word processor. There are a huge number of jobs available for those who can work with a lot of data as well, while there are a growing number of jobs for professionals who can manage thousands of computers working together - their maintenance, security, etc. Okay, so you have enough information to start your adventures into learning programming! You might hear people talk a lot about professionals losing jobs due to automation. Don’t let something like that be the reason behind why you want to learn how to program. Computer Science and programming has become more ingrained in school education, and our little ones are being coached to be industry ready. Always remember, programming is not everyone’s cup of tea and you shouldn’t do it just because everyone else is. Do it if you’re really passionate about solving problems in a better way. You will never know if programming is really meant for you until you try it. So go forth and get your hands dirty with some code! What is the difference between functional and object oriented programming? The Top 7 Python programming books you need to read Top 5 programming languages for crunching Big Data effectively

Python needs no introduction. It’s one of the top rated and growing programming languages, mainly because of its simplicity and wide applicability to solve a range of problems. Developers like yourself, beginners and experts alike, are looking to skill themselves up with Python. So I thought I would put together a list of Python programming books that I think are the best for learning Python - whether you're a beginner or experienced Python developer. Books for beginning to learn Python Learning Python, by Fabrizio Romano What the book is about This book explores the essentials of programming, covering data structures while showing you how to manipulate them. It talks about control flows in a program and teaches you how to write clean and reusable code. It reveals different programming paradigms and shows you how to optimize performance as well as debug your code effectively. Close to 450 pages long, the content spans twelve well thought out chapters. You’ll find interesting content on Functions, Memory Management and GUI app development with PyQt. Why Learn from Fabrizio Fabrizio has been creating software for over a decade. He has a master's degree in computer science engineering from the University of Padova and is also a certified Scrum master. He has delivered talks at the last two editions of EuroPython and at Skillsmatter in London. The Approach Taken The book is very easy to follow, and takes an example driven approach. As you end the book, you will be able to build a website in Python. Whether you’re new to Python or programming on the whole, you’ll have no trouble at all in following the examples. Download Learning Python FOR FREE. Learning Python, by Mark Lutz What the book is about This is one of the top most books on Python. A true bestseller, the book is perfectly fit for both beginners to programming, as well as developers who already have experience working with another language. Over 1,500 pages long, and covering content over 41 chapters, the book is a true shelf-breaker! Although this might be a concern to some, the content is clear and easy to read, providing great examples wherever necessary. You’ll find in-depth content ranging from Python syntax, to Functions, Modules, OOP and more. Why Learn from Mark Mark is the author of several Python books and has been using Python since 1992. He is a world renowned Python trainer and has taught close to 260 virtual and on-site Python classes to roughly 4,000 students. The Approach Taken The book is a great read, complete with helpful illustrations, quizzes and exercises. It’s filled with examples and also covers some advanced language features that recently have become more common in modern Python. You can find the book here, on Amazon. Intermediate Python books Modern Python Cookbook, by Steven Lott What the book is about Modern Python Cookbook is a great book for those already well versed with Python programming. The book aims to help developers solve the most common problems that they’re faced with, during app development. Spanning 824 pages, the book is divided into 13 chapters that cover solutions to problems related to data structures, OOP, functional programming, as well as statistical programming. Why Learn from Steven Steven has over 4 decades of programming experience, over a decade of which has been with Python. He has written several books on Python and has created some tutorial videos as well. Steven’s writing style is one to envy, as he manages to grab the attention of the readers while also imparting vast knowledge through his books. He’s also a very enthusiastic speaker, especially when it comes to sharing his knowledge. The Approach Taken The book takes a recipe based approach; presenting some of the most common, as well as uncommon problems Python developers face, and following them up with a quick and helpful solution. The book describes not just the how and the what, but the why of things. It will leave you able to create applications with flexible logging, powerful configuration, command-line options, automated unit tests, and good documentation. Find Modern Python Cookbook on the Packt store. Python Crash Course, by Eric Matthes What the book is about This one is a quick paced introduction to Python and assumes that you have knowledge of some other programming language. This is actually somewhere in between Beginner and Intermediate, but I've placed it under Intermediate because of its fast-paced, no-fluff-just-stuff approach. It will be difficult to follow if you’re completely new to programming. The book is 560 pages long and is covered over 20 chapters. It covers topics ranging from the Python libraries like NumPy and matplotlib, to building 2D games and even working with data and visualisations. All in all, it’s a complete package! Why Learn from Eric Eric is a high school math and science teacher. He has over a decade’s worth of programming experience and is a teaching enthusiast, always willing to share his knowledge. He also teaches an ‘Introduction to Programming’ class every fall. The Approach Taken The book has a great selection of projects that caters to a wide range of audience who’re planning to use Python to solve their programming problems. It thoughtfully covers both Python 2 and 3. You can find the book here on Amazon. Fluent Python, by Luciano Ramalho What the book is about The book is an intermediate guide that assumes you have already dipped your feet into the snake pit. It takes you through Python’s core language features and libraries, showing you how to make your code shorter, faster, and more readable at the same time. The book flows over almost 800 pages, with 21 chapters. You’ll indulge yourself in topics on the likes of Functions as well as objects, metaprogramming, etc. Why Learn from Luciano Luciano Ramalho is a member of the Python Software Foundation and co-founder of Garoa Hacker Clube, the first hackerspace in Brazil. He has been working with Python since 1998. He has taught Python web development in the Brazilian media, banking and government sectors and also speaks at PyCon US, OSCON, PythonBrazil and FISL. The Approach Taken The book is mainly based on the language features that are either unique to Python or not found in many other popular languages. It covers the core language and some of its libraries. It has a very comprehensive approach and touches on nearly every point of the language that is pythonic, describing not just the how and the what, but the why. You can find the book here, on Amazon. Advanced Python books The Hitchhiker's Guide to Python, by Kenneth Reitz & Tanya Schlusser What the book is about This isn’t a book that teaches Python. Rather, it’s a book that shows experienced developers where, when and how to use Python to solve problems. The book contains a list of best practices and how to apply these practices in real-world python projects. It focuses on giving great advice about writing good python code. It is spread over 11 chapters and 338 pages. You’ll find interesting topics like choosing an IDE, how to manage code, etc. Why Learn from Kenneth and Tanya Kenneth Reitz is a member of the Python Software Foundation. Until recently, he was the product owner of Python at Heroku. He is a known speaker at several conferences. Tanya is an independent consultant who has over two decades of experience in half a dozen languages. She is an active member of the Chicago Python User’s Group, Chicago’s PyLadies, and has also delivered data science training to students and industry analysts. The Approach Taken The book is highly opinionated and talks about what the best tools and techniques are to build Python apps. It is a book about best practices and covers how to write and ship high quality code, and is very insightful. The book also covers python libraries/frameworks that are focused on capabilities such as data persistence, data manipulation, web, CLI, and performance. You can get the book here on Amazon. Secret Recipes of the Python Ninja, by Cody Jackson What the book is about Now this is a one-of-a-kind book. Again, this one is not going to teach you about Python Programming, rather it will show you tips and tricks that you might not have known you could do with Python. In close to 400 pages, the book unearth secrets related to the implementation of the standard library, by looking at how modules actually work. You’ll find interesting topics on the likes of the CPython interpreter, which is a treasure trove of secret hacks that not many programmers are aware of, the PyPy project, as well as explore the PEPs of the latest versions to discover some interesting hacks. Why Learn from Cody Cody Jackson is a military veteran and the founder of Socius Consulting, an IT and business management consulting company. He has been involved in the tech industry since 1994. He is a self-taught Python programmer and also the author of the book series Learning to Program Using Python. He’s always bubbling with ideas and ways about improving the way he codes and has brilliantly delivered content through this book. The Approach Taken Now this one is highly opinionated too - the idea is to learn the skills from a Python Ninja. The book takes a recipe based approach, putting a problem before you and then showing you how you can wield Python to solve it. Whether you’re new to Python or are an expert, you’re sure to find something interesting in the book. The recipes are easy to follow and waste no time on lengthy explanations. You can find the book here on Amazon and here on the Packt website. So there you have it. Those were my top 7 books on Python Programming. There are loads of books available on Amazon, and quite a few from Packt that you can check out, but the above are a list of those that are a must-have for anyone who’s developing in Python. Read Next What are data professionals planning to learn this year? Python, deep learning, yes. But also… Python web development: Django vs Flask in 2018 Why functional programming in Python matters: Interview with best selling author, Steven Lott What the Python Software Foundation & Jetbrains 2017 Python Developer Survey had to reveal

Operator overloading is a form of polymorphism. Some operators change behaviors on different types. The classic example is the operator plus (+). On numeric values, plus is a sum operation and on String is a concatenation. Operator overloading is a useful tool to provide your API with a natural surface. Let's say that we're writing a Time and Date library; it'll be natural to have the plus and minus operators defined on time units.  In this article, we'll understand how Operator Overloading works in Kotlin. This article has been extracted from the book, Functional Kotlin, by Mario Arias and Rivu Chakraborty.  Kotlin lets you define the behavior of operators on your own or existing types with functions, normal or extension, marked with the operator modifier: class Wolf(val name:String) { operator fun plus(wolf: Wolf) = Pack(mapOf(name to this, wolf.name to wolf)) } class Pack(val members:Map<String, Wolf>) fun main(args: Array<String>) { val talbot = Wolf("Talbot") val northPack: Pack = talbot + Wolf("Big Bertha") // talbot.plus(Wolf("...")) } The operator function plus returns a Pack value. To invoke it, you can use the infix operator way (Wolf + Wolf) or the normal way (Wolf.plus(Wolf)). Something to be aware of about operator overloading in Kotlin—the operators that you can override in Kotlin are limited; you can't create arbitrary operators. Binary operators Binary operators receive a parameter (there are exceptions to this rule—invoke and indexed access). The Pack.plus extension function receives a Wolf parameter and returns a new Pack. Note that MutableMap also has a plus (+) operator: operator fun Pack.plus(wolf: Wolf) = Pack(this.members.toMutableMap() + (wolf.name to wolf)) val biggerPack = northPack + Wolf("Bad Wolf") The following table will show you all the possible binary operators that can be overloaded: Operator Equivalent Notes x + y x.plus(y) x - y x.minus(y) x * y x.times(y) x / y x.div(y) x % y x.rem(y) From Kotlin 1.1, previously mod. x..y x.rangeTo(y) x in y y.contains(x) x !in y !y.contains(x) x += y x.plussAssign(y) Must return Unit. x -= y x.minusAssign(y) Must return Unit. x *= y x.timesAssign(y) Must return Unit. x /= y x.divAssign(y) Must return Unit. x %= y x.remAssign(y) From Kotlin 1.1, previously modAssign. Must return Unit. x == y x?.equals(y) ?: (y === null) Checks for null. x != y !(x?.equals(y) ?: (y === null)) Checks for null. x < y x.compareTo(y) < 0 Must return Int. x > y x.compareTo(y) > 0 Must return Int. x <= y x.compareTo(y) <= 0 Must return Int. x >= y x.compareTo(y) >= 0 Must return Int. Invoke When we introduce lambda functions, we show the definition of Function1: /** A function that takes 1 argument. */ public interface Function1<in P1, out R> : Function<R> { /** Invokes the function with the specified argument. */ public operator fun invoke(p1: P1): R } The invoke function is an operator, a curious one. The invoke operator can be called without name. The class Wolf has an invoke operator: enum class WolfActions { SLEEP, WALK, BITE } class Wolf(val name:String) { operator fun invoke(action: WolfActions) = when (action) { WolfActions.SLEEP -> "$name is sleeping" WolfActions.WALK -> "$name is walking" WolfActions.BITE -> "$name is biting" } } fun main(args: Array<String>) { val talbot = Wolf("Talbot") talbot(WolfActions.SLEEP) // talbot.invoke(WolfActions.SLEEP) } That's why we can call a lambda function directly with parenthesis; we are, indeed, calling the invoke operator. The following table will show you different declarations of invoke with a number of different arguments: Operator Equivalent Notes x() x.invoke() x(y) x.invoke(y) x(y1, y2) x.invoke(y1, y2) x(y1, y2..., yN) x.invoke(y1, y2..., yN) Indexed access The indexed access operator is the array read and write operations with square brackets ([]), that is used on languages with C-like syntax. In Kotlin, we use the get operators for reading and set for writing. With the Pack.get operator, we can use Pack as an array: operator fun Pack.get(name: String) = members[name]!! val badWolf = biggerPack["Bad Wolf"] Most of Kotlin data structures have a definition of the get operator, in this case, the Map<K, V> returns a V?. The following table will show you different declarations of get with a different number of arguments: Operator Equivalent Notes x[y] x.get(y) x[y1, y2] x.get(y1, y2) x[y1, y2..., yN] x.get(y1, y2..., yN) The set operator has similar syntax: enum class WolfRelationships { FRIEND, SIBLING, ENEMY, PARTNER } operator fun Wolf.set(relationship: WolfRelationships, wolf: Wolf) { println("${wolf.name} is my new $relationship") } talbot[WolfRelationships.ENEMY] = badWolf The operators get and set can have any arbitrary code, but it is a very well-known and old convention that indexed access is used for reading and writing. When you write these operators (and by the way, all the other operators too), use the principle of least surprise. Limiting the operators to their natural meaning on a specific domain, makes them easier to use and read in the long run. The following table will show you different declarations of set with a different number of arguments: Operator Equivalent Notes x[y] = z x.set(y, z) Return value is ignored x[y1, y2] = z x.set(y1, y2, z) Return value is ignored x[y1, y2..., yN] = z x.set(y1, y2..., yN, z) Return value is ignored Unary operators Unary operators don't have parameters and act directly in the dispatcher. We can add a not operator to the Wolf class: operator fun Wolf.not() = "$name is angry!!!" !talbot // talbot.not() The following table will show you all the possible unary operators that can be overloaded: Operator Equivalent Notes +x x.unaryPlus() -x x.unaryMinus() !x x.not() x++ x.inc() Postfix, it must be a call on a var, should return a compatible type with the dispatcher type, shouldn't mutate the dispatcher. x-- x.dec() Postfix, it must be a call on a var, should return a compatible type with the dispatcher type, shouldn't mutate the dispatcher. ++x x.inc() Prefix, it must be a call on a var, should return a compatible type with the dispatcher type, shouldn't mutate the dispatcher. --x x.dec() Prefix, it must be a call on a var, should return a compatible type with the dispatcher type, shouldn't mutate the dispatcher. Postfix (increment and decrement) returns the original value and then changes the variable with the operator returned value. Prefix returns the operator's returned value and then changes the variable with that value. Now you know how Operator Overloading works in Kotlin. If you found this article interesting and would like to read more, head on over to get the whole book, Functional Kotlin, by Mario Arias and Rivu Chakraborty. Extension functions in Kotlin: everything you need to know Building RESTful web services with Kotlin Building chat application with Kotlin using Node.js, the powerful Server-side JavaScript platform

A lot of things make working in tech difficult. Technical debt is one of them. Whether you're working in-house or for an external team, you've probably experienced some tough challenges when it comes to legacy software. Most people have encountered strange internal software systems, a CMS that has been customized in a way that no one has the energy to fathom. Working your way around and through these can be a headache to say the least. In this year's Skill Up survey, we found that Technical debt and legacy issues are seen by developers as the biggest barrier to business goals. According to 49% of respondents, old technology and software is stopping organizations from reaching their full potential. But it might also be stopping developers from moving forward in their careers. Read the report in full. Sign up to our newsletter and download the PDF for free. Technical debt and the rise of open source Arguably, issues around technical debt have become more pronounced in the last decade as the pace of technical change has seemingly increased. I say seemingly, because it's not so much that we're living in an entirely new technical landscape. It's more that the horizons of that landscape are expanding. There are more possibilities and options open to businesses today. Technology leadership is difficult in 2018. To do it well, you need to stay on top of new technologies. But you also need a solid understanding of your internal systems, your team, as well as wider strategic initiatives and business goals. There are a lot of threads you need to manage. Are technology leaders struggling with technical debt? Perhaps technology leaders are struggling. But perhaps they're also making the best of difficult situations. When you're juggling multiple threads in the way I've described, you need to remain focused on what's important. Ultimately, that's delivering software that delivers value. True, your new mobile app might not be ideal; the internal CMS you were building for a client might not offer an exemplary user experience. But it still does the job - and that, surely is the most important thing? We can do better - let's solve technical debt together It's important to be realistic. In the age of burn out and over work, let's not beat ourselves up when things aren't quite what we want. Much of software engineering is, after all, making the best of a bad situation. But the solutions to technical debt can probably be found in a cultural shift. The lack of understanding of technology on the part of management is surely a large cause of technical debt. When projects aren't properly scoped and when deadlines are set without a clear sense of what level of work is required, that's when legacy issues begin to become a problem. In fact, it's worth looking at all the other barriers. In many ways, they are each a piece of the puzzle if we are to use technology more effectively - more imaginatively - to solve business problems. Take these three: Lack of quality training or learning Team resources Lack of investment in projects All of these point to a wider cultural problem with the way software is viewed in businesses. There's no investment, teams are under-resourced, and support to learn and develop new skills is simply not being provided. With this lack of regard for software, it's unsurprising that developers are spending more time solving problems on, say, legacy code, than solving big, interesting problems. Ones that might actually have a big impact. One way of solving technical debt, then, is to make a concerted effort to change the cultural mindset. Yes, some of this will need to come from senior management, but all software engineers need to take responsibility. This means better communication and collaboration, a commitment to documentation - those things that are so easy to forget to do well when you could be shipping code. What happens if we don't start solving technical debt Technical debt is like global warming - it's happening already. We feel the effects every day. However, it's only going to get worse. Yes, it's going to damage businesses, but it's also going to hurt developers. It's restricting the scope of developers to do the work they want to do and make a significant impact on their businesses. It seems as though we're locked in a strange cycle where businesses talk about the importance of 'digital skills' and technical knowledge gaps but ironically can't offer the resources or scope for talented developers to actually do their job properly. Developers bring skills, ideas, and creativity to jobs only to find that they're isn't really time to indulge that creativity. "Maybe next year, when we have more time" goes the common refrain. There's never going to be more time - that's obvious to anyone who's ever had a job, engineer or otherwise. So why not take steps to start solving technical debt now? Read next 8 Reasons why architects love API driven architecture Python, Tensorflow, Excel and more – Data professionals reveal their top tools The best backend tools in web development

Python Software Foundation together with Jetbrains conduct their developer survey every year and at the end of 2017, over 9,500 developers from all over the world participated in this insightful Python developer survey. The learnings are pretty interesting and so, I thought I’d quickly summarise the most relevant points here. So here we go. TL;DR: Adoption of Python 3 is growing rapidly, but 25% of developers are yet to migrate to Python 3 despite the closely looming deadline (1st of Jan, 2020). There are as many Python developers doing primarily data science as there are those focused on web development. This is quite different from the 2016 findings, where there was a difference of 17% between Web and Data. A majority of Python developers use JavaScript, HTML/CSS and SQL along with Python. Django, Pandas, NumPy and Matplotlib are the most popular python frameworks. Jupyter Notebook and Docker are the most popular technologies used along with Python. Among cloud platforms, AWS is the most popular. Both editions of PyCharm (Community and Professional) are the most popular tools for Python development, followed by SublimeText. Code autocompletion, code refactorings and writing unit tests are the most widely used features in Python. More than half the respondents had a full-time job, working on Python. A majority of respondents held the role of a Developer/Programmer and belonged to the age group 21-39. Most of the developers are located in the US, India and China. The above stats are quite interesting no doubt. This got me thinking about the why behind those numbers. Here’s my perspective on some of those findings. I would love to hear yours in the comments section below. How is Python being used? Starting with the usage of Python, the survey revealed that close to 80% of the respondents used Python as their primary language for development. When asked which languages they generally used it with, the top responses were JavaScript, HTML/CSS and SQL. On the other hand, a lot of Java developers and those using Bash/shell, seem to use Python as their secondary language. This shows that Python is quite interoperable with a number of other languages, making it versatile to use in web, enterprise and server side scripting. Now when it comes to what tasks Python is used for in day to day development, it wasn’t a surprise when respondents mentioned Data Analysis. More than 50% use Python as their primary language for data analysis, however, only 32% claimed that they used it for Machine Learning. On the other hand, 54% mentioned that they used it for web development. 36% responded that they used Python for DevOps and system administration purposes. This isn’t surprising as most developers tend to stick to a particular tool/stack as far as possible. Developers also responded that they used Python the most for Web Development, apart from anything else, with Machine Learning + Data Analysis close on its heels. Most DevOps and Sys admins use Python as their secondary language - that might be because shell/bash are their primary languages. In the 2016 survey, the percentage of web developers was much more than ML/Data Analysts, but the difference has reduced greatly. What roles do these developers hold? When asked what roles these developers hold, the responses were quite interesting! While nearly a quarter were in a combination of Data Analysis and Machine Learning roles, another quarter were in a combination of Data Analysis and Web Development! 15% claimed to be in Web Development and Machine Learning. This relationship, although quite unlikely, is extremely interesting and worth exploring further. One reason could be that developers are building machine learning solutions that are offered to customers as a web application, rather than as a desktop application. Another reason could also be that a lot of web apps these days are becoming more data driven and require some kind of machine learning components running under the hood. What version of Python are developers rolling with and what tools do they use it with? A very surprising fact that surfaced from the survey was that 25% of developers still haven’t migrated their codebases to Python 3 and are still working with Python 2. This is quite astonishing, since the support for Python 2 will be discontinued in less than two years (from Jan 1, 2020 to be precise). Although, the adoption for Python 3 has been growing steadily over the years, most of the developers who were still using Python 2 turned out to be web developers. This is so because data scientists might have moved into using Python quite recently, as compared to web developers who might have been using Python for a long time and hence, haven’t migrated their legacy code. What are their prefered tool set with Python? When asked about the tools that developers used, the web developers responded that a majority of them used Django(76%), while 53% used Requests and 49% used Flask. When it came to GUI frameworks, 9% of developers used PyQT / PyGTK / wxPython while 6% used TkInter. 29% of these developers mentioned that they used scientific libraries like NumPy / pandas / Matplotlib / scipy. This is quite supportive of the overlap between both the GUI development and Data Science roles. On the other hand, Data Scientists responded that 65% used NumPy / pandas / Matplotlib / scipy. 38% used Keras / Theano / TensorFlow / scikit-learn, while 31% and 27% used Django and Flask respectively. Django was a clear winner in the tools section, with an overall of 41% developers using it. When asked about what tools they used along with Python, the web developers responded that 47% used Docker, 46% used an ORM like SQLAlchemy, PonyORM, etc. and 40% used Redis. 27% of them used Jupyter Notebook. The Data Scientists on the other hand, used Jupyter Notebook a lot (52%). 40% of them used Anaconda and 23% Docker. Of the various cloud platforms, developers chose AWS the most (65%). When it came to Python features that were used the most, Code autocompletion (84%), code refactorings (82%) and writing unit tests (81%), made the top of the list. 75% developers used SQL databases while only 46% used NoSQL. Of the various IDEs and Editors, PyCharm in both its versions, community and professional, was the most popular, closely tailed by Sublime, Vim, IDLE, Atom, and VS Code. While Web Developers preferred PyCharm, data scientists prefer Jupyter Notebook. Developer Profile: Employment, Job Roles and Experience Unsurprisingly, 52% of Python developers claimed that they were in a full-time job. This ties in well with the 2018 StackOverflow Developer survey which labeled Python as the “most wanted” programming language. So developers out there, if you’re well versed with Python, you’re more likely to be hired. Talking about job roles, 73% held the role of a Developer/Programmer, while 19% held the role of a Data Analyst and 18% an Architect. Interestingly, 7% of them held the role of a CIO / CEO / CTO. In terms of years of experience, the results were well balanced with almost as many developers having more than 5 years of experience as those with less than 5 years of experience. 67% of the respondents were in the age group of 21-39, meaning that a majority of young folk seem to be using Python. If you’re one of them, and are looking to progress in your career, check out our extensive catalog of Python titles. As for geographic location of the developers, 18% were from the US while 13% were from India and 7% from China. Should you move to Python 3? 7 Python experts’ opinions Python experts talk Python on Twitter: Q&A Recap Introducing Dask: The library that makes scalable analytics in Python easier

“We’re Agile”. That’s the kind of phrase I would expect from a football team, a troupe of ballet dancers or maybe a martial artist. Everytime I hear it come from the mouth of a software professional, I go like “Oh boy, not again!”. So here I am to talk about something that might touch a nerve or two, of an Agile fan. I’m talking about whether you should be abandoning agile once and for all! Okay, so what is Agile? Agile software development is an approach to software development, where requirements and solutions evolve through a collaborative effort of self-organizing and cross-functional teams, as well as the end user. Agile advocates adaptive planning, evolutionary development, early delivery, and a continuous improvement. It also encourages a rapid and flexible response to change. The Agile Manifesto was created by some of the top software gurus on the likes of Uncle Bob, Martin Fowler, et al. The values that it stands for are: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan Apart from these, it follows 12 principles, as given here, through which it aims to improve software development. At its heart, it is a mindset. So what’s wrong? Honestly speaking, everything looks rosy from the outside until you’ve actually experienced it. Let me ask you at this point, and I’d love to hear your answers in the comments section below. Has there never been a time when you felt at least one of the 12 principles were a hindrance to your personal, as well as team’s development process? Well, if yes, you’re not alone. But before throwing the baby out with the bathwater, let’s try and understand a bit and see if there’s been some misinterpretation, which could be the actual culprit. Here are some common misinterpretations of what it is, what it can and cannot do. I like to call them: The 7 Deadly Sins #1 It changes processes One of the main myths about Agile is that it changes processes. It doesn't really change your processes, it changes your focus. If you’ve been having problems with your process and you feel Agile would be your knight in shining armor, think again. You need something more than just Agile and Lean. This is one of the primary reasons teams feel that Agile isn’t working for them - they’ve not understood whether they should have gone Agile or not. In other words, they don’t know why they went Agile in the first place! #2 Agile doesn’t work for large, remote teams The 4th point of the Agile manifesto states, “developers must work together daily throughout the project”. Have you ever thought about how “awesome aka impractical” it is to coordinate with teams in India, all the way from the US on a daily basis? The fact is that it’s not practically possible for such a thing to happen when teams are spread across time zones. What it intends is to have the entire team communicating with each other on a daily basis and there’s always the possibility of a Special Point of Contact to communicate and pass on the information to other team members. So no matter how large the team, if implemented in the right way, Agile works. Strong communication and documentation helps a great deal here. #3 Prefer the “move fast and break things” approach Well, personally I prefer to MFABT. Mostly because at work, I’m solely responsible for my own actions. What about when you’re part of a huge team that’s working on something together? When you take such an approach, there are always hidden costs of being 'wrong'. Moreover, what if everytime you moved fast, all you did was break things? Do you think your team’s morale would be uplifted? #4 Sprints are counterproductive People might argue that sprints are dumb and what’s the point of releasing software in bits and pieces? I think what you should actually think about is whether what you’re focusing on can actually be done quicker. Faster doesn’t apply to everything. Take making babies for example. Okay, jokes apart, you’ll realise you might often need to slow things down in order to go fast, so that you reach your goal without making mistakes. At least not too many costly ones anyway. Before you dive right into Agile, understand whether it will add value to what you do. #5 I love micromanagement Well, too bad for you dude, Agile actually promotes self-driven, self-managed and autonomous teams that are learning continuously to adapt and adjust. In enterprises where there is bureaucracy, it will not work. Bear in mind that most organizations (may be apart from startups) are hierarchical in nature which brings with bureaucracy in some form or flavor. #6 Scrum saves time Well, yes it does. Although if you’re a manager and think Scrum is going to cut you a couple of hours from paying attention to your individual team members, you’re wrong. The idea of Scrum is to identify where you’ve reached, what you need to do today and whether there’s anything that might get in the way of that. Scrum doesn’t cover for knowing your team members problems and helping them overcome them. #7 Test everything, everytime No no no no…. That’s a wrong notion, which in fact wastes a lot of time. What you should actually be doing is automated regression tests. No testing is bad too; you surely don’t want bad surprises before you release! Teams and organisations tend to get carried away by the Agile movement and try to imitate others without understanding whether what they’re doing is actually in conjunction with what the business needs. Now back to what I said at the beginning - when teams say they’re agile, half of them only think they are. It was built for the benefit of software teams all across the globe, and from what teams say, it does work wonders! Like any long term relationship, it takes conscious efforts and time everyday to make it work. Should you abandon Agile? Yes and no. If you happen to have the slightest hint that one or more of the following are true for your organisation, you really need to abandon Agile or it will backfire: Your team is not self-managed and lacks matured and cross-functional developers Your customers need you to take approvals at every release stage Not everyone in your organisation believes in Agile Your projects are not too complex Always remember, Agile is not a tool and if someone is trying to sell you a tool to help you become Agile, they’re looting you. It is a mindset; a family that trusts each other, and a team that communicates effectively to get things done. My suggestion is to go ahead and become agile, only if the whole family is for it and is willing to transform together. In other words, Agile is not a panacea for all development projects. Your choice of methodology will come down to what makes the best sense for your project, your team and your organization. Don’t be afraid to abandon agile in favor of new methodologies such as Chaos Engineering and MOB Programming or even go back to the good ol’ waterfall model. Let us know what you think of Agile and how well your organisation has adapted to it, if has adopted it. You can look up some fun discussions about whether it works or sucks on Hacker news: In a nutshell, why do a lot of developers dislike Agile? Poor Man’s Agile: Scrum in 5 Simple Steps What is Mob Programming? 5 things that will matter in application development in 2018 Chaos Engineering: managing complexity by breaking things

What is Odoo? Odoo is an all-in-one management software that offers a range of business applications. It forms a complete suite of enterprise management applications targeting companies of all sizes. It is versatile in the sense that it can be used across multiple categories including CRM, website, e-commerce, billing, accounting, manufacturing, warehouse, and project management, and inventory. The community version is free-of-charge and can be installed with ease. Odoo is one of the fastest growing open source, business application development software products available. With the announcement of version 11 of Odoo, there are many new features added to Odoo and the face of business application development with Odoo has changed. In Odoo 11, the online installation documentation continues to improve and there are now options for Docker installations. In addition, Odoo 11 uses Python 3 instead of Python 2.7. This will not change the steps you take in installing Odoo but will change the specific libraries that will be installed. While much of the process is the same as previous versions of Odoo, there have been some pricing changes in Odoo 11. There are only two free users now and you pay for additional users. There is one free application that you can install for an unlimited number of users, but as soon as you have more than one application, then you must pay $25 for each user, including the first user. If you have thought about developing in Odoo, now is the best time to start. Before I convince you on why Odoo is great, let’s take a step back and revisit our fundamentals. What is an ERP? ERP is an acronym often used for Enterprise Resource Planning. The ERP gives a global and real-time view of data that can enable companies to address concerns and drive improvements. It automates the core business operations such as the order to fulfillment and procures to pay processes. It also reduces risk management for companies and enhances customer services by providing a single source for billing and relationship tracking. Why Odoo? Odoo is Extensible and easy to customize Odoo's framework was built with extensibility in mind. Extensions and modifications can be implemented as modules, to be applied over the module with the feature being changed, without actually changing it. This provides a clean and easy-to-control and customized applications. You get integrated information Instead of distributing data throughout several separate databases, Odoo maintains a single location for all the data. Moreover, the data remains consistent and up to date. Single reporting system Odoo has a unified and single reporting system to analyze and track the status. Users can also run their own reports without any help from IT. Single reporting systems, such as those provided by Odoo ERP software helps make reporting easier and customizable. Built around Python Odoo is built using the Python programming language, which is one of the most popular languages used by developers. Large community The capability to combine several modules into feature-rich applications, along with the open source nature of Odoo, is probably the important factors explaining the community that has grown around Odoo. In fact, there are thousands of community modules available for Odoo, covering virtually every topic, and the number of people getting involved has been steadily growing every year. Go through our video, Odoo 11 development essentials to learn to scaffold a new module, create new models, and use the proper functions that make Odoo 11 the best ERP out there. Top 5 free Business Intelligence tools How to build a live interactive visual dashboard in Power BI with Azure Stream Tableau 2018.1 brings new features to help organizations easily scale analytics

Architecture Decision Records (ADRs) document all the decisions made about software. Every change is recorded in a plain text file sitting inside a version control system (like GitHub). The record should be a complement to the information you can find in a version control system. The ADR provides context and information around every decision made about a piece of software. Why are lightweight Architecture Decision Records needed? We are always making decisions when we build software. Even the simplest piece of software will have required the engineer to take a number of different decisions. Often these decisions aren't obvious. If you've ever had to work with code written by someone else you're probably familiar with this sort of situation. You might have even found that when you come across someone else's code, you need to make a further decision. Either you can simply accept what has been written, and merely surmise and assume why it has been done in the way that it, or you can decide to change it, based on your own judgement. Neither option is ideal. This was what Michael Nygard identified in this blog post in 2011. This was when the concept of Architecture Decision Records first emerged. An ADR should prevent situations like this arising. That makes life easier for you. More importantly, it should mean that every decision is transparent to everyone involved. So, instead of blindly accepting something or immediately changing it, you can simply check the Architecture Decision Record. This will then inform how you proceed. Perhaps you need to make a change. But perhaps you also now understand the context of why something was built in the way it was. Any questions you might have should be explicitly answered in the architecture decision record. So, when you start asking yourself why has she done it like that? instead of floundering helplessly, you can find the answer in the ADR. Why lightweight Architecture Decision Records now? Architecture Decision Records aren't a new thing. Nygard wrote his post all the way back in 2011, after all. But the fact remains that the context from which Nygard was writing in 2011 was very specific. Today it is mainstream. As we've moved away from monolithic architecture towards microservices or serverless, decision making has become more and more important in software engineering. This is a point that is well explained in a blog post here: "The rise of lean development and microservices... complicates the ability to communicate architecture decisions. While these concepts are not inherently opposed to documentation, their processes often fail to effectively capture decision-making processes and reasoning. Another possible inefficiency when recording decisions is bad or out-of-date documentation. It's often a herculean effort to keep large, complex architecture documents current, making maintenance one of the most common barriers to entry." ADRs are, then, a way of managing the complexity in modern software engineering. They are a response to a fundamental need to better communicate decisions. Most importantly, they codify decision-making within the development process. It is when they are lightweight and sit within the project itself that they are most effective. Architecture Decision Record template Architecture Decision Records must follow a template. Not only does that mean everyone is working off the same page, it also means people are actually more likely to document their decisions. Think about it: if you're asked to note how you decide to do something without any guidelines, you're probably not going to do it at all. Below, you'll find an Architecture Decision Record example template. There are a number of different templates you can use, but it's probably best to sit down with your team and agree on what needs to be captured. An Architecture Decision Record example Date Decision makers [who was involved in the decision taken] Category [which part of the architecture does this decision pertain to] Contextual outline [Explain why this decision was made. Outline the key considerations and assumptions at play] Impact consequences [What does this decision mean for the project? What should someone reading this be aware of in terms of future decisions?] As I've already noted, there are a huge number of ways you may want to approach this. Use this as a starting point. Read next Enterprise Architecture Concepts Reactive Programming and the Flux Architecture

If you're on the Packt Hub, it's likely that you already know how to code. But perhaps you don't - and if you don't, it's our job to let you know why you should learn to code. And even if you do know how, evangelizing and explaining to others why they should learn is incredibly important. People often think of writing code as something very specialized and technically demanding. There’s sometimes even a perception of people that do write code as a certain kind of person. But this is, luckily, demonstrably untrue. Writing code is something that can be done by anyone. While there are some incredibly specialized roles that require incredibly detailed levels of programming knowledge, in actual fact a huge number of roles today use software in creative and analytical ways. That’s why learning how to code is so essential. Let’s take a look at 5 reasons why you should learn how to write code. Learn to code to better understand how software works Code is everywhere. Learning how to write it is a way of understanding how so much of the world works. Perhaps you’re a writer. Or maybe a designer. Maybe you work in marketing. Whatever you’re doing, it’s likely that you’re working in a digital environment. Understanding how these are built and how they work puts you at an advantage, in that you have an extra dimension of knowledge of how things work ‘under the hood’. Learning to code will help you better manipulate and use data The world runs on data. Code can help you better understand it and manage it. Excel is a sad part of business life. You don’t have to look had to finding someone complaining about spreadsheets. But if you’ve ever been impressed with someone’s dexterity with Excel, that’s really just a type of code. In the world of data today there are far more sophisticated systems for processing and managing data. But if you’re working with large datasets in any capacity, learning how to write SQL scripts, for example, is a great place to begin your coding adventure. Learn to code simply to have fun Coding is fun and creative! Before talk of spreadsheets and databases turns you off, it’s important to note that writing code and learning to program is fun. If you’re just starting out seeing the effect of your code as you build your first web page is a really neat experience. A lot of people think coding is purely technical - but the technical part is just a small aspect of it. Really, it’s about solving problems and being creative. Even if you’re not sure how learning to code might help you professionally right now, having a creative outlet might just inspire your future. Coding isn't as hard as you think It’s not actually that hard… Writing code isn’t really that hard. It might take some getting used to, but once you’re over those first steps you’ll feel like an expert. True, learning how to write code always involves some ups and downs, and if you’re getting deeper into more complex programming concepts and tasks, it will be challenging. But that doesn’t mean that it’s beyond you. It’s a valuable skill you can actually develop quickly. You can learn to code without spending lots of money It’s a valuable skill you can develop cheaply. Learning how to write code could increase your earning power. Even if it doesn’t transform your career, it could still give you a push to get to the next step. But while development often requires expensive training and certifications that your employer might pay for, learning the basics of coding is something you can do without spending very much money at all. There’s lots of free resources out there. And even the very best resources won’t put you out of pocket. Slow down to learn how to code faster